Problem with generating pixel representation of image upload in Adobe XD

[Stavros]

Hey guys,

im currently developing a plugin for Adobe XD.

At some point the extension will access an image uploaded, and convert it to a pixel representation for further processing.

For this code out of the npm package get-image-pixel is used, shown below:

/****************************** Code Block *********************************/**  
module.exports = function(image, opts) {

   opts = opts || {}
   opts.x = opts.x || 0
   opts.y = opts.y || 0
   opts.width = typeof opts.width === 'number' ? opts.width : image.width
   opts.height = typeof opts.height === 'number' ? opts.height : image.height

   if (!context) {
      canvas = document.createElement("canvas")
      context = canvas.getContext('2d')
   }

   canvas.width = opts.width
   canvas.height = opts.height
   context.clearRect(0, 0, opts.width, opts.height)
   context.drawImage(image, opts.x, opts.y, opts.width, opts.height, 0, 0, opts.width, opts.height)

   var imgData

   try {
       imgData = context.getImageData(0, 0, opts.width, opts.height)
   }
   catch(e) {
      module.exports.dispose()
      throw e
   }

   return imgData.data
}

/****************************** Code Block *********************************/

The problem i’m facing now is when trying to get the context of the created canvas (indicated by the little arrow in the Code Block).

The thrown problem is:

Plugin TypeError: canvas.getContext is not a function

Interestingly enough the created canvas is a owner document of the following form rather than a canvas element:

My guess is that it has something to do with the adobe xd environment and the creating of the canvas element is not working as

it would in a browser.

My guess is that it has something to do the t

{ _ownerDocument: 

   { _ownerDocument: [Circular],

     _selectorEngine: 

      { _document: [Circular],
        _styleSheetList: {},
        _styleNodes: {},
        _styleSheets: [],
        _styleNodesMutation: 0,
        _linkNodesMutation: 0,
        _cachedNodeList: null,
        _cachedEnclosingClassNames: {}, 

..... 

Do you have any idea how to solve this problem or even another way to get the pixel representation of an image? You would really help me out!

[Velara]

The XD environment doesn’t support all of the features of a browser.

There is a feature request for adding a BitmapData object here.

I can’t seem to find it now but there is somewhere some examples of loading a remote image, saving it locally and then reading it as a byte array.

[stevekwak]

@Stavros
This tutorial might help.
https://adobexdplatform.com/plugin-docs/tutorials/how-to-make-network-requests/

[Stavros]

Thanks for your response! Although this sounds like something that could work, performing the image conversion into a byte array somewhere else then sending it to the user, I would prefer to do it locally on the XD instance of the user.
Maybe the support of some browser features would make it easier for plugin development.
Are you guys at Adobe thinking about that ?

[peterflynn]

If I’m understanding correctly, you have an image file which your plugin has downloaded from a server, and you want to bring that image into the user’s XD document? In that case, there’s no need to deal with its raw pixel data or to do any kind of image drawing yourself. Just save the image file to a temp folder and then create an ImageFill based on that file (assuming it’s a format XD supports, like PNG or JPG).

After downloading the image file (using XHR or fetch()), you can follow the example at the top of the ImageFill documentation except use your downloaded file in place of the getFileForOpening() filepicker call shown in that example.

[Stavros]

No the image file was uploaded by the user already. When he has different images in his boar the plugin is able to access the one which was selected. For further analysis the plugin needs to convert the image to raw pixel data. No server is involved.
With the get-image.pixel package (https://www.npmjs.com/package/get-image-pixels) should be done.

The problem occurs when creating a canvas and trying to access its context.
Specifically at the line marked with a little error:
–> context = canvas.getContext(‘2d’)

[kerrishotts]

The error occurs specifically because the execution environment does not have 2D canvas support. It’s on our backlog, but I have no timeframe for implementation.

The only workaround currently is to parse the image into an ArrayBuffer manually (there are libraries out there that can do this), do the desired pixel manipulation, and then go from there. It’s not as fast, but is an option. (This is how plugins like Stark work.)

[stevekwak]

@Stavros

Thanks for your response! Although this sounds like something that could work, performing the image conversion into a byte array somewhere else then sending it to the user, I would prefer to do it locally on the XD instance of the user.
Maybe the support of some browser features would make it easier for plugin development.
Are you guys at Adobe thinking about that ?

How to make network requests · Adobe XD Plugin Reference now reflects the modified ImageFill API. The example now saves the binary file and loads the file directly.

[Stavros]

@kerrishotts

manipulation

Hey, im trying to access the content of my uploaded and selected image in the artboard of Adobe XD through ImageFill.

The ImageFill documentation shows how to access for instance the height and weight of the image. Is there a way to access the actual pixel and rgb values?

[stevekwak]

@Stavros an ImageFill object is a representation of a locally saved file or a base64 string. So if you want to get the actual pixels, you will have to extract it from the file itself. Hope this helps!

[Stavros]

@stevekwak

it from the file itself

Hey Steve, thank you for your quick reply! Yes but how can this happen inside my plugin? The only thing I really have is the selected rectangle with the image is ImageFill.
Or is there another way to access files in the Adobe XD Artboard?

[stevekwak]

What’s the output you need exactly? Is it base64? arraybuffer? or a PNG file?

[Stavros]

I have to access the single pixels of the image and their rgb values. Usually I would do this using canvas but AdobeXD does not offer that functionality yet in it’s environment. Is Arraybuffer the way to go and if yes how does one access the information through the ImageFill? Is it through a property?

[stevekwak]

@Stavros unfortunately, ImageFill property does not have a property for that. You would have to first create a rendition of a given node (image) on the document and access the file to get arraybuffer of the image like below:

const data = await rendition.read({ format: require("uxp").storage.formats.binary });

This will give you the arraybuffer you need.

[Stavros]

@stevekwak Do I have to store the rendition file locally on a folder chosen by the user?

[stevekwak]

You don’t have to. You can use the temp directory - https://adobexdplatform.com/plugin-docs/reference/uxp/storage-index.html#getting-access-to-the-local-file-system

[saud]

Hey steve, how can i get the array buffer here? I’m trying to upload images to a nodejs server.

const exportRendition = async (selection, documentRoot) => {
	const folder = await fs.getTemporaryFolder();
	let artboardList = [];
	await documentRoot.children.forEach(e => {
		artboardList.push(e);
	});
	const artboardChildren = artboardList.filter(node => node instanceof scenegraph.Artboard);
	const arr = await artboardChildren.reduce(async (all, ab) => {
		const currentTask = ab.children.map(async item => {
			const file = await folder.createFile(`${item.guid}.png`, { overwrite: true });
			const obj = {};
			obj.node = item;
			obj.outputFile = file;
			obj.type = 'png';
			obj.scale = 1;
			return obj;
		});
		const current = await Promise.all(currentTask);
		return (await all).concat(current);
	}, []);
	const renditions = await Promise.all(arr);
	await application.createRenditions(renditions);

	const entries = await folder.getEntries();
	const renderedEntries = [];
	const obj = {};
	let flag = false;
	entries.forEach(entry => {
		for (let i = 0; i < entry.name.length; i++) {
			obj.name = entry.name;
			obj.nativePath = entry.nativePath;
		}
		renderedEntries.push(obj);
		flag = true;
	});

	if (flag && renderedEntries) {
		console.log(flag, renderedEntries);
		const bufferImages = [];
		entries.forEach(image => {
			bufferImages.push(image.read({ format: require('uxp').storage.formats.binary }));
			console.log(bufferImages, 'ddddd');
		});
	}
};

[stevekwak]

Remeber to await when you read images like

await image.read({ format: require('uxp').storage.formats.binary })

[saud]

	if (flag && renderedEntries) {
		console.log(flag, renderedEntries);
		const bufferImages = [];
		entries.forEach(async image => {
			bufferImages.push(await image.read({ format: require('uxp').storage.formats.binary }));
			console.log(bufferImages, 'ddddd');
		});
	}
};

Still returning empty object.
renderedEntries look like this before the buffer image function

[ { name: 'fbec6339-f72e-43ac-bb64-2b302e90fb6e.png',
    nativePath: 'C:\\Users\\Saud\\AppData\\Local\\Packages\\Adobe.CC.XD_adky2gkssdxte\\TempState\\UXP\\PluginsTemp\\External\\d31194ea\\fbec6339-f72e-43ac-bb64-2b302e90fb6e.png' },
  { name: 'fbec6339-f72e-43ac-bb64-2b302e90fb6e.png',
    nativePath: 'C:\\Users\\Saud\\AppData\\Local\\Packages\\Adobe.CC.XD_adky2gkssdxte\\TempState\\UXP\\PluginsTemp\\External\\d31194ea\\fbec6339-f72e-43ac-bb64-2b302e90fb6e.png' },
  { name: 'fbec6339-f72e-43ac-bb64-2b302e90fb6e.png',
    nativePath: 'C:\\Users\\Saud\\AppData\\Local\\Packages\\Adobe.CC.XD_adky2gkssdxte\\TempState\\UXP\\PluginsTemp\\External\\d31194ea\\fbec6339-f72e-43ac-bb64-2b302e90fb6e.png' },
  { name: 'fbec6339-f72e-43ac-bb64-2b302e90fb6e.png',
    nativePath: 'C:\\Users\\Saud\\AppData\\Local\\Packages\\Adobe.CC.XD_adky2gkssdxte\\TempState\\UXP\\PluginsTemp\\External\\d31194ea\\fbec6339-f72e-43ac-bb64-2b302e90fb6e.png' },
  { name: 'fbec6339-f72e-43ac-bb64-2b302e90fb6e.png',
    nativePath: 'C:\\Users\\Saud\\AppData\\Local\\Packages\\Adobe.CC.XD_adky2gkssdxte\\TempState\\UXP\\PluginsTemp\\External\\d31194ea\\fbec6339-f72e-43ac-bb64-2b302e90fb6e.png' },
  { name: 'fbec6339-f72e-43ac-bb64-2b302e90fb6e.png',
    nativePath: 'C:\\Users\\Saud\\AppData\\Local\\Packages\\Adobe.CC.XD_adky2gkssdxte\\TempState\\UXP\\PluginsTemp\\External\\d31194ea\\fbec6339-f72e-43ac-bb64-2b302e90fb6e.png' },
  { name: 'fbec6339-f72e-43ac-bb64-2b302e90fb6e.png',
    nativePath: 'C:\\Users\\Saud\\AppData\\Local\\Packages\\Adobe.CC.XD_adky2gkssdxte\\TempState\\UXP\\PluginsTemp\\External\\d31194ea\\fbec6339-f72e-43ac-bb64-2b302e90fb6e.png' },
  { name: 'fbec6339-f72e-43ac-bb64-2b302e90fb6e.png',
    nativePath: 'C:\\Users\\Saud\\AppData\\Local\\Packages\\Adobe.CC.XD_adky2gkssdxte\\TempState\\UXP\\PluginsTemp\\External\\d31194ea\\fbec6339-f72e-43ac-bb64-2b302e90fb6e.png' },
  { name: 'fbec6339-f72e-43ac-bb64-2b302e90fb6e.png',
    nativePath: 'C:\\Users\\Saud\\AppData\\Local\\Packages\\Adobe.CC.XD_adky2gkssdxte\\TempState\\UXP\\PluginsTemp\\External\\d31194ea\\fbec6339-f72e-43ac-bb64-2b302e90fb6e.png' } ]

so the renditions are present in the image buffer function

[stevekwak]

You might actually get the arraybuffer back but not logged out with console.log. Try converting it to base64 string and log it out by using this function

function base64ArrayBuffer(arrayBuffer) {
    var base64 = ''
    var encodings = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/'

    var bytes = new Uint8Array(arrayBuffer)
    var byteLength = bytes.byteLength
    var byteRemainder = byteLength % 3
    var mainLength = byteLength - byteRemainder

    var a, b, c, d
    var chunk

    // Main loop deals with bytes in chunks of 3
    for (var i = 0; i < mainLength; i = i + 3) {
        // Combine the three bytes into a single integer
        chunk = (bytes[i] << 16) | (bytes[i + 1] << 8) | bytes[i + 2]

        // Use bitmasks to extract 6-bit segments from the triplet
        a = (chunk & 16515072) >> 18 // 16515072 = (2^6 - 1) << 18
        b = (chunk & 258048) >> 12 // 258048   = (2^6 - 1) << 12
        c = (chunk & 4032) >> 6 // 4032     = (2^6 - 1) << 6
        d = chunk & 63               // 63       = 2^6 - 1

        // Convert the raw binary segments to the appropriate ASCII encoding
        base64 += encodings[a] + encodings[b] + encodings[c] + encodings[d]
    }

    // Deal with the remaining bytes and padding
    if (byteRemainder == 1) {
        chunk = bytes[mainLength]

        a = (chunk & 252) >> 2 // 252 = (2^6 - 1) << 2

        // Set the 4 least significant bits to zero
        b = (chunk & 3) << 4 // 3   = 2^2 - 1

        base64 += encodings[a] + encodings[b] + '=='
    } else if (byteRemainder == 2) {
        chunk = (bytes[mainLength] << 8) | bytes[mainLength + 1]

        a = (chunk & 64512) >> 10 // 64512 = (2^6 - 1) << 10
        b = (chunk & 1008) >> 4 // 1008  = (2^6 - 1) << 4

        // Set the 2 least significant bits to zero
        c = (chunk & 15) << 2 // 15    = 2^4 - 1

        base64 += encodings[a] + encodings[b] + encodings[c] + '='
    }

    return base64
}