CN115134603B

CN115134603B - Image encoding, decoding method, device, electronic device and storage medium

Info

Publication number: CN115134603B
Application number: CN202110315542.8A
Authority: CN
Inventors: 赵登; 夏超
Original assignee: Alibaba Innovation Co
Current assignee: Alibaba Innovation Co
Priority date: 2021-03-24
Filing date: 2021-03-24
Publication date: 2025-03-14
Anticipated expiration: 2041-03-24
Also published as: CN115134603A

Abstract

The embodiment of the application provides an image coding method, an image decoding method, an image coding device, an electronic device and a storage medium, wherein the image coding method comprises the steps of determining a display level of a display layer in a screen image, determining a top display layer in the screen image according to the display level, determining a first display area interested by a user in the screen image at least based on the top display layer, setting the compressed image quality of the first display area to be higher than the compressed image quality of a second display area in the screen image, wherein the second display area is a display area outside the first display area in the screen image, and coding the screen image according to the compressed image quality of the first display area and the compressed image quality of the second display area to obtain a coded image. The embodiment of the application can improve the image coding efficiency under the condition of considering the image quality and the image compression rate.

Description

Image encoding and decoding method and device, electronic equipment and storage medium

Technical Field

The embodiment of the application relates to the technical field of image processing, in particular to an image encoding and decoding method, an image encoding and decoding device, electronic equipment and a storage medium.

Background

The image coding is a technology for representing image content with smaller data volume by image compression, and has wide application in various fields such as image editing, image transmission and the like, wherein the image quality and the image compression rate are two important indexes of the image coding technology, the higher the image quality is, the clearer the coded image is, the higher the image compression rate is, the smaller the data volume of the coded image is, so the image coding technology mainly reduces the data volume of the image by improving the image compression rate as much as possible under the condition of meeting the image quality, and in the background, how to combine the image quality and the image compression rate and efficiently finish the image coding is a problem which needs to be solved by a person skilled in the art.

Disclosure of Invention

In view of the above, embodiments of the present application provide an image encoding method, an image decoding method, an image encoding device, an image decoding device, an electronic device, and a storage medium, so as to improve image encoding efficiency while considering both image quality and image compression rate.

In order to achieve the above object, the embodiment of the present application provides the following technical solutions:

an image encoding method applied to an electronic device, the method comprising:

Determining a display level of a display layer in the screen image;

determining a top display layer in the screen image according to the display level, and determining a first display area in the screen image which is interested by a user at least based on the top display layer;

setting the compressed image quality of the first display area to be higher than that of a second display area in the screen image, wherein the second display area is a display area outside the first display area in the screen image;

And encoding the screen image according to the compressed image quality of the first display area and the compressed image quality of the second display area to obtain an encoded image.

The embodiment of the application also provides an image decoding method which is applied to the electronic equipment and comprises the following steps:

Acquiring a coded image corresponding to a screen image, wherein the compressed image quality of a first display area which is interested by a user in the coded image is higher than that of a second display area, the second display area is a display area outside the first display area, and the first display area at least comprises a top display layer in the screen image;

decoding the encoded image.

The embodiment of the application also provides an image coding device, which comprises:

the display level determining module is used for determining the display level of the display layer in the screen image;

the interest area determining module is used for determining a top display layer in the screen image according to the display level, and determining a first display area which is interested by a user in the screen image at least based on the top display layer;

The compression setting module is used for setting the compression image quality of the first display area to be higher than the compression image quality of a second display area in the screen image, wherein the second display area is a display area outside the first display area in the screen image;

And the encoding module is used for encoding the screen image according to the compressed image quality of the first display area and the compressed image quality of the second display area to obtain an encoded image.

The embodiment of the application also provides an image coding device, which comprises an encoder, wherein the encoder is used for:

The method comprises the steps of determining a display level of a window in a screen image, determining a top-layer window in the screen image according to the display level, determining a first display area which is interested by a user in the screen image at least based on the top-layer window, setting the compressed image quality of the first display area to be higher than the compressed image quality of a second display area in the screen image, wherein the second display area is a display area outside the first display area in the screen image, and encoding the screen image according to the compressed image quality of the first display area and the compressed image quality of the second display area to obtain an encoded image.

The embodiment of the application also provides an image decoding device, which comprises:

The coding image acquisition module is used for acquiring a coding image corresponding to a screen image, wherein the compression image quality of a first display area interested by a user in the coding image is higher than that of a second display area, the second display area is a display area outside the first display area, and the first display area at least comprises a top display layer in the screen image;

And the decoding module is used for decoding the coded image.

The embodiment of the application also provides electronic equipment, which comprises at least one memory and at least one processor, wherein the memory stores one or more computer executable instructions, and the processor calls the one or more computer executable instructions to execute the image coding method or the image decoding method.

Embodiments of the present application also provide a storage medium storing one or more computer-executable instructions that, when executed, implement an image encoding method as described above, or an image decoding method as described above.

The image coding method provided by the embodiment of the application can code the screen image, aiming at the display layer in the screen image, the embodiment of the application can acquire the display layer of the display layer, and the top display layer in the screen image is determined according to the display layer, because the user is interested in the top display layer in the screen image with high probability, the embodiment of the application can determine the first display area interested by the user in the screen image at least based on the top display layer, thus, aiming at the first display area in the screen image and the second display area outside the first display area, the embodiment of the application can set the compressed image quality of the first display area to be higher than the compressed image quality of the second display area, so that the first display area has higher image quality, and the image data amount of the second display area is reduced (namely, the image data amount of the second display area is reduced), and further, the embodiment of the application can code the screen image according to the compressed image quality of the first display area and the compressed image quality of the second display area, and the code rate of the image is improved on the premise of ensuring that the whole image quality is reduced.

Meanwhile, the first display area of interest of the user in the screen image is determined at least based on the top display layer, namely the embodiment of the application can determine the first display area of interest of the user based on the display level of the display layer in the screen image without identifying the area of interest of the user in the screen image by using an ROI identification technology, thereby greatly improving the determination efficiency of the part of interest of the user in the screen image and realizing image coding in an efficient manner. Therefore, the embodiment of the application can improve the image coding efficiency under the condition of considering the image quality and the image compression efficiency.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings that are required to be used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only embodiments of the present application, and that other drawings can be obtained according to the provided drawings without inventive effort for a person skilled in the art.

FIG. 1 is a flowchart of an image encoding method according to an embodiment of the present application;

FIG. 2 is an exemplary diagram of a display layer;

FIG. 3 is a flowchart of determining a first display area according to an embodiment of the present application;

FIG. 4 is another exemplary diagram of a display layer;

FIG. 5 is a diagram of another example of a display layer;

FIG. 6 is a diagram of yet another example of a display layer;

Fig. 7 is a component architecture diagram of an electronic device according to an embodiment of the present application;

fig. 8 is a flowchart of an image decoding method according to an embodiment of the present application;

FIG. 9 is an exemplary diagram of a desktop sharing service;

Fig. 10 is a block diagram of an image encoding apparatus according to an embodiment of the present application;

FIG. 11 is a block diagram of an image encoding apparatus according to an embodiment of the present application;

Fig. 12 is another block diagram of an image encoding apparatus according to an embodiment of the present application;

fig. 13 is a further block diagram of an image encoding apparatus according to an embodiment of the present application;

fig. 14 is a block diagram of an electronic device according to an embodiment of the present application;

fig. 15 is a block diagram of an image decoding apparatus according to an embodiment of the present application.

Detailed Description

The following description of the embodiments of the present application will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.

An image coding scheme of the current mainstream is to identify an ROI (Region Of Interest ) region of interest of a user and a non-ROI region of non-interest of the user from an image based on an ROI (Region Of Interest ) identification technology, so that a higher compressed image quality is set for the ROI region, a lower compressed image quality is set for the non-ROI region, and then the image is compressed to finish image coding; the higher the compressed image quality is, the larger the corresponding image data amount is, the lower the compressed image quality is, and the corresponding image data amount is smaller; the image coding scheme applying the ROI identification technology can enable the image quality of the ROI area to be higher, but the image quality of the non-ROI area to be lower, namely, on the premise of meeting the image quality requirement of the ROI area, the image code rate of the non-ROI area is reduced, the whole image data volume is reduced, and the effect of improving the image compression rate is achieved;

Although the above image coding scheme using the ROI identification technology can ensure the image quality of the ROI area of interest to the user and improve the image compression rate by reducing the image code rate of the non-ROI area of not interest to the user on the basis of identifying the ROI area and the non-ROI area of the image, there is still a certain limitation, specifically:

The ROI identification technology needs to be implemented based on a machine learning technology or an image identification technology, and the machine learning technology or the image identification technology has complex processing logic, so when image coding is performed on an image with specific image content, it may not be a good choice to implement image coding by applying the ROI identification technology; for example, in the case of encoding a screen image, there is a display layer (a display layer such as a window or the like) in the screen image, and particularly when there are a plurality of display layers in the screen image, since the display layer may block part of the image content, and the machine learning technique or the image recognition technique itself is a relatively complex processing logic, the use of the machine learning technique or the image recognition technique to recognize the detailed image content in the screen image as an ROI region and the non-detailed image content as a non-ROI region may require higher system resources, which makes the image encoding scheme applying the ROI recognition technique unable to efficiently complete the image encoding in the case of the screen image having the display layer, i.e., has a limitation in the image encoding efficiency of the image encoding scheme applying the ROI recognition technique in the case of the screen image having the display layer facing the screen image;

Based on this, the inventors of the embodiments of the present application propose a novel image encoding scheme in the case where a screen image is encoded and a display layer is present in the screen image, so as to improve image encoding efficiency while taking both image quality and image compression efficiency into consideration.

As an optional implementation, fig. 1 shows an optional flow of an image encoding method provided by an embodiment of the present application, where the image encoding method may be performed by an electronic device, where the electronic device may be an electronic device provided with a screen image, and a user may implement a screen operation based on the screen image provided by the electronic device;

in a possible application scenario, the electronic device may be a terminal device, for example, after the terminal device may encode a local screen image displayed locally, an encoded image (such as a video encoded image in a video form) may be formed, where the encoded image may be stored locally in the terminal device and may also be shared with other devices, and in a possible application example, a screen sharing service is taken as an example, the terminal device may encode the local screen image displayed locally, and the scheme provided by the embodiment of the present application may encode the local screen image to form a video encoded image, where the video encoded image may be shared with other terminal devices through a network or a cloud service that provides a screen sharing service;

In other possible application scenarios, the electronic device may be a cloud service device, for example, when the cloud service device provides a cloud screen service for the terminal device, the cloud service device may utilize the scheme provided by the embodiment of the present application to encode a cloud screen image to form a video encoded image, and the video encoded image may be transmitted to the terminal device, so that the terminal device may implement a cloud screen operation after decoding and displaying the video image;

optionally, referring to fig. 1, the image encoding method provided in the embodiment of the present application may include:

Step S100, determining a display level of a display layer in the screen image.

Optionally, the screen image may be a local screen image corresponding to a local screen of the terminal device, and accordingly, the image encoding method provided by the embodiment of the present application may be executed by the terminal device to encode the local screen image displayed by the local screen of the terminal device, and in other optional implementations, the screen image may be a cloud screen image corresponding to a cloud screen service of the cloud service device, and accordingly, the image encoding method provided by the embodiment of the present application may be executed by the cloud service device to encode the cloud screen image.

In the embodiment of the application, whether the screen image is a local screen of the terminal equipment or a cloud screen image corresponding to a cloud screen service, the screen image can be provided with a display layer, the number of the display layers can be one or more, and the display layer can be understood as a component used for displaying image content in the screen image, such as a window, an icon and the like used for displaying the image content in the screen image. It will be appreciated that the image content of the screen image may include, in addition to the image content of each display layer, image content located outside the display layer, such as image content of a screen background, for example, taking the screen image as a desktop image, the display layer as a window, and the image content of the desktop image includes, in addition to the image content of each window, image content of a desktop background.

In a possible implementation, the display layer system of the electronic device (whether the terminal device or the cloud service device) may be used to manage the display layers in the screen, through which information such as a display level, a position, a resolution and the like of each display layer in the screen may be obtained, so that the display layer system of the embodiment of the present application may determine at least the display level of the display layer in the screen image, and it may be understood that if only one display layer is in the screen image, the display level of the display layer is single, and the one display layer is located at the top layer, and if a plurality of display layers are in the screen image, the plurality of display layers have a plurality of display levels, and the display layer located at the last display level is superimposed on the display layer located at the next display level;

in a more specific example, taking a screen image as a desktop image and a display layer as a window, a window system of the electronic device may be used to manage the windows in the screen, so that information such as a display level, a position, a resolution and the like of each window can be obtained through the window system.

And step S110, determining a top display layer in the screen image according to the display level, and determining a first display area in the screen image, which is interested by a user, at least based on the top display layer.

According to the research of the inventor of the embodiment of the application, when a user operates a screen, an interested display layer (such as a window) is placed on the top layer, so that the user can conveniently edit data, watch content and the like through the display layer on the top layer, therefore, after the display level of the display layer in the screen image is determined, the top layer display layer (for convenience of explanation, the display layer on the top layer in the screen image can be called as the top layer display layer) on the top layer in the screen image can be determined according to the display level, and therefore, the display area corresponding to the top layer display layer should be at least used as a first display area (for convenience of explanation, the display area of interest of the user in the screen image can be called as the first display area) in the screen image, and it can be understood that the first display area of interest of the user in the screen image needs to ensure image quality in the encoding process.

In an optional implementation, the embodiment of the present application may use only the display area corresponding to the top display layer as the first display area of interest to the user in the screen image, where no matter whether the display layer in the screen image is one or more, that is, no matter whether the display layer in the screen image is one or more, the embodiment of the present application uses only the display area corresponding to the top display layer as the first display area after determining the top display layer.

In another alternative implementation, the first display area of interest to the user in the screen image may include other display areas besides the display area corresponding to the top display layer, for example, in the case that there are multiple display layers in the screen image, the other display areas of interest to the user may relate to display layers different from the top display layer, such as the next display layer of the top display layer, or one or more display layers designated by the user in addition to the top display layer;

In this example, the display layers 1,2 and 3 (in this example, the display layer may be understood as, for example, a window) are displayed in the screen in addition to the display area corresponding to the top display layer 1, wherein, based on the display layers 1,2 and 3, the display layer 1 is the top display layer, the display layer 2 is lower than the display layer 1 in one stage, and is the next display layer of the display layer 1, while the display layer 3 is lower than the display layer 2 in one stage, in the example of fig. 2, the first display area of interest of the user in the screen image may also include the display area corresponding to the next display layer 2 of the top display layer in addition to the display area corresponding to the top display layer 1;

In other optional implementations, the target display layer may be specified by a specific operation or a shortcut setting operation of the user, that is, in the case where the default top display layer is the display layer of interest to the user, the embodiment of the present application further allows the user to specify the target display layer of interest, for example, when the user opens a certain display layer, the user may specify the opened display layer as the display layer of interest through a specific operation (such as a specific operation of adding a mark of interest or the like) or the shortcut setting operation.

Step S120, setting the compressed image quality of the first display area higher than the compressed image quality of a second display area in the screen image, where the second display area is a display area outside the first display area in the screen image.

After determining the first display area in the screen image, since the user is interested in the image content of the first display area with a high probability, a higher compressed image quality needs to be set for the first display area, and for the second display area outside the first display area in the screen image, since the user is not interested in the image content of the second display area with a high probability, a lower compressed image quality can be set for the second display area, so that in the image encoding process, the image code rate of the second display area is reduced, and the image data amount of the second display area is reduced.

As an alternative implementation, in a lossy coding algorithm of an image (such as a video image), a quantization coefficient is related to coding quality and coding rate of the image, and is one of factors determining image quality, in image coding, the quantization coefficient is a sequence number of a quantization step, the quantization coefficient and quantization loss of the image form a positive correlation, and form a negative correlation with the image quality, that is, the smaller the value of the quantization coefficient is, the finer the quantized coefficient of the quantization step is, the smaller the quantization loss of the image is, the higher the image quality is, otherwise, the larger the value of the quantization coefficient is, the coarser the quantized coefficient of the quantization step is, the larger the quantization loss of the image is, that is, the lower the image quality is, that is, the quantization coefficient and the image quality form a negative correlation;

Based on this, the embodiment of the application can set quantization coefficients for the first display area and the second display area to realize that compressed image quality is set for the first display area and the second display area, that is, the embodiment of the application can set quantization coefficients for the first display area and lower than quantization coefficients for the second display area, so that the whole image data amount after screen image coding is reduced for the first display area which is interested by a user and the second display area which is not interested by the user with large probability outside the first display area, and the effect of improving the image compression rate is achieved by setting quantization coefficients for the first display area and lower than quantization coefficients for the second display area, so that the image quality of the image content of the first display area is higher than the image quality of the image content of the second display area.

Of course, setting the quality of the compressed image is only an alternative means by setting the quantization coefficients, and other parameters that can embody the quality of the compressed image may also be supported by embodiments of the present application.

In an alternative implementation, if the screen image has only one display layer, the display layer is a top display layer, the display area corresponding to the top display layer can be directly used as the first display area, and the second display area except for the first display area in the screen image can be regarded as the background area of the screen image;

In another alternative implementation, if there are multiple display layers in the screen image, the multiple display layers may include a top display layer, a target display layer that is interesting to the user except for the top display layer, and a non-target display layer that is not interesting to the user, for example, as shown in connection with the example of fig. 2, the screen image exists in display layers 1, 2 and 3, display layer 1 is a top display layer, display layer 2 is a target display layer that is interesting to the user except for the top display layer, display layer 3 is a non-target display layer, then the display areas corresponding to display layer 1 and display layer 2 are used as the first display area, then the display area corresponding to display layer 3 and the background area of the screen image are used as the second display area, in which case, the quantization coefficients of the top display layer and the target display layer may be set to be the same, and lower than the quantization coefficients of the non-target display layer and the background area in the screen image;

It should be noted that, the background area of the screen image may be regarded as a display area other than the display layers in the screen image, for example, in the example of fig. 2, the display areas other than the display layers 1, 2, and 3 may be regarded as the background area of the screen image.

And step S130, encoding the screen image according to the compressed image quality of the first display area and the compressed image quality of the second display area to obtain an encoded image.

In an alternative implementation, the embodiment of the application can encode the screen image based on the quantization coefficient of the first display area and the quantization coefficient of the second display area to obtain the encoded image corresponding to the screen image, thereby completing the image encoding work of the screen image.

In a further alternative implementation, the encoded image may be stored by an electronic device that performs image encoding, or may be transmitted by the electronic device to another device, so as to decode the encoded image on the other device and display the decoded image;

In a possible application scenario, taking a screen sharing service used by a terminal device as an example, sharing a screen to other terminal devices, after the terminal device encodes a local screen image corresponding to the local screen by using the scheme provided by the embodiment of the application, the terminal device can share the encoded image to other terminal devices participating in the screen sharing service, so that the screen sharing among a plurality of terminal devices is realized with lower image data volume;

in other possible application scenarios, taking the cloud service device as an example of providing the cloud screen service for the terminal device, the cloud service device can utilize the scheme provided by the embodiment of the application to encode the cloud screen image, and transmit the encoded image to the terminal device using the cloud screen service, so that the cloud screen service is provided for the terminal device with lower image data volume.

It should be noted that, the embodiment of the present application has the advantage of higher image coding efficiency in addition to the effect of improving the image compression rate on the premise of guaranteeing the image quality of the first display area, specifically, the embodiment of the present application determines the first display area of interest of the user in the screen image based on at least the top display layer in the screen image by determining the display level of the display layer in the screen image, so that in the case that the display layer exists in the screen image, the embodiment of the present application can determine the portion of interest of the user in the screen image based on at least the display level of the display layer in the screen image, without identifying the region of interest of the user in the screen image by means of the ROI identification technology, thereby reducing the system resources required for determining the portion of interest of the user in the screen image, and further improving the image coding efficiency.

In view of the fact that there are multiple display layers in the screen image, a description will be given below of possible implementation manners of determining the first display area in the screen image that is of interest to the user according to the embodiment of the present application.

As an optional implementation, fig. 3 illustrates an optional flow of determining the first display area provided by an embodiment of the present application, where, as shown in fig. 3, the flow may include:

Step S200, determining whether a target display layer of interest to the user exists in the screen image except the top display layer, if so, executing step S210, and if not, executing step S220.

Step S210, determining the display areas corresponding to the top display layer and the target display layer as the first display area of interest to the user in the screen image.

Step S220, determining the display area corresponding to the top display layer as a first display area interested by the user in the screen image.

In the embodiment of the application, the display area corresponding to the top display layer is taken as the first display area which is interested by the user, whether the target display layer which is interested by the user exists in the display layer of the screen image can be further analyzed, so that when the target display layer which is interested by the user exists in the screen image, the display area corresponding to the target display layer can be taken as the first display area, and the image quality of the top display layer and the target display layer can be ensured in the image coding process;

In a possible implementation, the target display layer may be a next display layer of the top display layer, and the embodiment of the present application may analyze whether a target display layer of interest to the user exists in the screen image by analyzing whether the user is interested in the next display layer of the top display layer;

in other possible implementations, the user may specify the target display layer. The embodiment of the application can judge whether the target display layer which is appointed by the user and is interested in exists in the screen image. If the target display layer which is designated by the user and is interested in exists in the screen image, the embodiment of the application can determine the top display layer and the display area corresponding to the designated target display layer as the first display area.

In one example, the user-specified object of interest display layer is, for example, a user-added object display layer with an interest tag. The embodiment of the application can judge whether the target display layer added with the interesting label exists in the screen image except the top display layer. Therefore, when judging that the target display layer added with the interesting label exists in the screen image, the embodiment of the application can determine the display area corresponding to the top display layer and the target display layer added with the interesting label as the first display area.

More specifically, the target display layer is taken as the next display layer of the top display layer to describe, whether the user is interested in the next display layer is mainly to analyze whether the user is interested in the image content of the next display layer, and specifically, different analysis strategies can be provided based on different scenes of the screen (editing data and/or viewing content) operated by the user in combination with the top display layer and the next display layer, and only several possible cases of the user interested in the next display layer are further described below.

In a first case, the embodiment of the present application may determine, at least according to the display content of the next display layer, whether the user will be interested in the next display layer, for example, when the display content of the next display layer is the content of interest to the user, it may be determined that the user is interested in the next display layer, and at this time, the next display layer may be used as the target display layer; in a possible implementation, the display content of the next display layer is the content of interest to the user, for example, the type of the display content of the next display layer is the type of interest to the user (for example, by analyzing the user preference to determine whether the type of the display content of the next display layer is preferred by the user or not), and for example, the next display layer displays dynamic content (for example, video, animation and other dynamic content), since when displaying the dynamic content, the content of the next display layer has a large change in the time domain, it is possible for the user to set the display layer displaying the dynamic content as the next display layer of the top layer display layer, that is, the user may place the display layer displaying the dynamic content on the next display layer of the top layer intentionally, so that the user may pay attention to the dynamic content displayed on the next display layer when paying attention to the top layer display layer, where possible situations such as the top layer window is a chat window, the next window of the top layer window is a movie playing window, that is a user watching a movie while chatting, and other situations are possible;

For the first case, the embodiment of the application can judge whether the user is interested in the next display layer or not at least according to the display content of the next display layer; for example, in a possible implementation, the embodiment of the present application may determine whether the next display layer displays dynamic content through a display layer activity detection (e.g., window activity detection) function, so as to determine that the user is interested in the next display layer at least when the determination result of the display layer activity detection function is yes, where the display layer activity detection function may be understood as a function of detecting whether the content of the display layer changes with time;

In a further optional implementation, a situation that some more details may exist is that when determining that the next display layer displays dynamic content, the embodiment of the present application may directly determine that a user is interested in the next display layer, that is, as long as the next display layer displays dynamic content, the embodiment of the present application may directly determine that the user is interested in the next display layer no matter what other display attribute of the next display layer is, and as another optional implementation of a situation, the embodiment of the present application further needs to combine an area proportion of the next display layer that is not blocked when determining that the next display layer displays dynamic content to further determine whether the user is interested in the next display layer, which is specifically described as follows:

The embodiment of the application can further determine the area proportion of the unoccluded area of the next display layer, and judge whether the area proportion of the unoccluded area of the next display layer is not smaller than a set first proportion, thereby determining that the user is interested in the next display layer when the area proportion of the unoccluded area of the next display layer is not smaller than the first proportion, namely the area proportion of the unoccluded area of the next display layer is not smaller than the set first proportion, indicating that most of the area of the next display layer is not occluded, and the user is very likely to pay attention to the top display layer and the next display layer at the moment;

For example, in combination with the illustration of fig. 4, the window 1 is a top-level window, the window 2 is a next window, the window 2 displays dynamic content, since the area proportion of most of the window 2 is not blocked, the user pays attention to the window 2 when paying attention to the window 1, in order to reduce misjudgment of the region of interest of the user, the embodiment of the application can enlarge the region of interest of the user on the basis of the window 1 (top-level display layer), and consider that the user is also interested in the window 2, so that the embodiment of the application can use the display regions corresponding to the window 1 and the window 2 as the first display region;

it should be noted that the area ratio of the non-occluded area of the next display layer is understood as the area of the non-occluded area of the next display layer divided by the area of the next display layer, and the first ratio may be a predetermined area ratio of considering that most of the content of the display layer is non-occluded, for example, 60%, etc., although the numerical value herein is merely an exemplary, and the specific numerical value of the first ratio may be specifically set as the case may be.

In the second case, the embodiment of the application can judge whether the user is interested in the next display layer according to the area of the top display layer and the area relation between the top display layer and the next display layer, and can understand that if the area of the top display layer is smaller and the area of the next display layer is larger than that of the top display layer, from the sense angle of the user, the user generally cannot focus on the top display layer under the condition of scaling the area of the top display layer because the area of the top display layer is smaller, and the user is more likely to focus on the next display layer with the area larger than that of the top display layer, so that the next display layer can be used as the target display layer interested by the user, and possible situations of the case are that the top window is a music lyric window, the next window of the top window is a text window, namely the user listens to music while watching text and the like;

For the second case, the embodiment of the application can judge whether the user is interested in the next display layer or not according to at least the area of the top display layer and the area relation between the top display layer and the next display layer, for example, when the area of the top display layer is smaller and the area of the next display layer is larger than that of the top display layer, the next display layer is considered to be the target display layer interested by the user;

In a more specific alternative implementation, the embodiment of the present application may determine whether the area ratio of the top display layer to the screen image is not greater than the set second ratio, and whether the area ratio of the top display layer to the screen image is at least a first multiple of the area of the top display layer, so that when it is determined that the area ratio of the top display layer to the screen image is not greater than the set second ratio, and the area ratio of the next display layer to the top display layer is at least a first multiple of the area of the top display layer, the user is considered to be interested in the next display layer, and the next display layer may be the target display layer;

for example, in combination with fig. 5, window 1 is a top-layer window, window 2 is a next window, the area ratio of window 1 occupied by the screen image is smaller, and the area of window 2 is far greater than that of window 1, so that in order to reduce misjudgment of the user region of interest, the embodiment of the application can enlarge the region of interest of the user on the basis of window 1 (top-layer display layer), and consider that the user is also interested in window 2, so that the embodiment of the application can use the display regions corresponding to window 1 and window 2 as the first display region.

In the third case, the embodiment of the application can judge whether the user is interested in the next display layer of the top display layer by judging whether the top display layer is at the main body position of the screen image, and can determine that the user is interested in the next display layer of the top display layer if the top display layer is not at the main body position of the screen image; it should be noted that, the user will generally place the focused display layer at the main position of the screen (such as the central position of the screen) so as to facilitate the user to perform the screen operation, if the top display layer is not located at the main position of the screen (such as the top display layer is not located at the central position of the screen, and the top display layer may be located at the left upper, left lower, right upper, right lower, etc. corner positions of the screen), it is explained that the user may not place the main focus on the top display layer, and the area of interest of the user needs to be enlarged to reduce misjudgment of the area of interest of the user.

In an example, as shown in fig. 6, window 1 is a top window, window 2 is a next window, window 1 is located at the upper left corner of the screen image and is not located in the center position, so as to reduce erroneous judgment of the region of interest of the user, in the embodiment of the present application, the region of interest of the user can be enlarged based on window 1 (top display layer), and the user is considered to be interested in window 2, so that the display regions corresponding to window 1 and window 2 can be used as the first display region in the embodiment of the present application.

While several cases and analysis strategies are described above in which a user is interested in a next display layer of a top display layer, it should be noted that these cases and analysis strategies are merely optional cases in which a user is interested in the next display layer, and other possible cases in which a user is interested in the next display layer may be supported in embodiments of the present application, which are considered to be the scope of protection of the embodiments of the present application.

Of course, the next display layer of the top display layer is only an optional case of the target display layer of interest to the user, and the embodiment of the application can also support the case that the target display layer is designated by the user, and the like, and can also support the case that the user designates the target display layer and analyze whether the next display layer of the top display layer is the target display layer of interest to the user.

In the above, the possible implementation scheme of determining the first display area is described for the case that a plurality of display layers exist in the screen image, and it needs to be further described that, in the case that a plurality of display layers exist in the screen image, the embodiment of the application may not analyze whether a target display layer of interest to the user exists in the screen image, but directly take the display area corresponding to the top display layer as the first display area, that is, further analyze whether a target display layer of interest to the user exists in the screen image on the basis of the top display layer, so as to enlarge the display area of interest to the user is only an optional means.

After determining the first display area of interest to the user in the screen image, the range included in the first display area and the range included in the second display area may be different for different situations (the second display area is a display area other than the first display area in the screen image), as will be described in detail below.

In an optional implementation, when the screen image has only one display layer, the display area corresponding to the top display layer is used as the first display area, and the background area in the screen image is used as the second display area, so that when the quantized coefficient of the first display area is set and is lower than that of the second display area, the quantized coefficient of the top display layer can be set, and is lower than that of the background area, for example, if the first quantized coefficient corresponding to the ROI area and the second quantized coefficient corresponding to the non-ROI area are uniformly set, and the first quantized coefficient is lower than that of the second quantized coefficient, the first quantized coefficient can be set for the display area corresponding to the top display layer, and the second quantized coefficient can be set for the background area, so that the image content of the top display layer is encoded with the lower first quantized coefficient corresponding to the ROI area, and the image content of the background area is encoded with the higher second quantized coefficient corresponding to the non-ROI area.

In another alternative implementation, if the screen image has multiple display layers, if only the display area corresponding to the top display layer is used as the first display area, the display area corresponding to the non-top display layer and the background area in the screen image are used as the second display area, as shown in fig. 2, for example, if only the display layer 1 is used as the display layer of interest to the user, the display area corresponding to the display layer 1 may be used as the first display area, and the display areas corresponding to the display layers 2 and 3 and the background area may be used as the second display area, and for this case, the embodiment of the present application may set quantization coefficients in a plurality of ways:

In an alternative implementation, in the case of uniformly setting a first quantization coefficient corresponding to an ROI area and uniformly setting a second quantization coefficient corresponding to a non-ROI area, the embodiment of the present application may set the first quantization coefficient for a display area corresponding to a top-layer display layer, and set the second quantization coefficient for a display area corresponding to a non-top-layer display layer and a background area;

in other alternative implementations, if the quantized coefficients of the display layer of interest to the user are set to be lower than the quantized coefficients of the display layer of no interest to the user, and the quantized coefficients of the display layer of no interest to the user increase with decreasing display levels (i.e., for a plurality of display layers of no interest to the user, the quantized coefficients are in negative correlation with the display levels), the quantized coefficients of the top-layer display layer may be set to be lower than the quantized coefficients of the non-top-layer display layer, and the quantized coefficients of the non-top-layer display layer increase with decreasing display levels, and at the same time, the quantized coefficients of the background area are set to be not lower than the quantized coefficients of the display layer of the lowest display level, and in combination with the example shown in fig. 2, if only the display layer 1 is used as the display layer of interest to the user, the quantized coefficients of the display layer 1 are set to be lower than the quantized coefficients of the display layers 2 and 3, and the quantized coefficients of the display layer 2 are lower than the quantized coefficients of the display layer 3, and the quantized display of the background area is set to be not lower than the quantized coefficients of the display layer 3.

In yet another alternative implementation, where there are multiple display layers in the screen image, if the display layer of interest to the user includes a top display layer and a target display layer other than the top display layer, the multiple display layers in the screen image may be divided into three parts, namely a top display layer, the target display layer and a non-target display layer not of interest to the user, and as illustrated in connection with the example of fig. 2, if the display layer 2 is a target display layer of interest to the user in addition to the top display layer 1, the display layer 3 may be regarded as a non-target display layer;

Therefore, the display area corresponding to the top display layer and the target display layer in the screen image can be used as a first display area, the display area corresponding to the non-target display layer in the screen image and the background area can be used as a second display area, the first display area can be the display area corresponding to the display layers 1 and 2, and the second display area can be the display area corresponding to the display layer 3, as shown in the example of fig. 2, and in this case, the quantization coefficients can be set in various ways according to the embodiment of the application:

In an alternative implementation, in the case of uniformly setting a first quantization coefficient corresponding to an ROI area and uniformly setting a second quantization coefficient corresponding to a non-ROI area, the embodiment of the present application may set the first quantization coefficient for a display area corresponding to a top display layer and a target display layer, and set the second quantization coefficient for a display area corresponding to a non-target display layer and a background area;

In other alternative implementations, if the quantization coefficients of the display layer of interest to the user are set to be lower than the quantization coefficients of the display layer of no interest to the user, and the quantization coefficients of the display layer of no interest to the user increase with decreasing display levels (i.e., for a plurality of display layers of no interest to the user, the quantization coefficients are in negative correlation with the display levels), the embodiment of the application may set the quantization coefficients of the top display layer and the target display layer to be the same and lower than the quantization coefficients of the non-target display layer, and when the non-target display layer is a plurality of, the quantization coefficients of the non-target display layer increase with decreasing display levels, and at the same time, the quantization coefficients of the background region are set to be not lower than the quantization coefficients of the display layer of the lowest display level, as shown in the example of fig. 2, and in combination with the example of fig. 2, the quantization coefficients of the display layers 1 and 2 may be the same and lower than the quantization coefficients of the display layer 3, and if there are other display layers below the display layer, the quantization coefficients of the display layer 3 and the display layer below the quantization coefficients of the display layer are set to increase with decreasing display levels of the display layer.

In other possible implementations, the screen image may have only two display layers, i.e., a top display layer and a next display layer, and if both the top display layer and the next display layer are the display layers of interest to the user, the embodiment of the present application may set quantization coefficients of the display layers lower than quantization coefficients of the background region, e.g., set a first quantization coefficient corresponding to the ROI region for the display layer, and set a second quantization coefficient corresponding to the non-ROI region for the background region.

The possible situations of the first display area and the possible situations of the second display area are described above, and the possible ways of setting quantization coefficients for the first display area and the second display area are described, so that based on the foregoing description, the embodiment of the application can obtain the first display area with the lower quantization coefficient and the second display area with the higher quantization coefficient, further encode the screen image based on the quantization coefficients of the first display area and the quantization coefficients of the second display area, and can obtain the encoded image corresponding to the screen image, thereby completing the image encoding work of the screen image. Because the quantization coefficient and the image quality are in a negative correlation, in the encoding process of the screen image, the embodiment of the application can ensure the image quality of the first display area based on the quantization coefficient with a lower first display area, and can reduce the image code rate of the second display area under the condition of losing a certain image quality of the second display area based on the quantization coefficient with a higher second display area, so that the image data volume of the second display area is reduced, and the embodiment of the application can reduce the overall data volume of the encoded image and improve the image compression rate on the premise of ensuring the image quality of the first display area. Meanwhile, the first display area of interest of the user in the screen image is determined at least based on the top display layer, namely, the embodiment of the application can determine the first display area of interest of the user based on the display level of the display layer in the screen image without identifying the area of interest of the user in the screen image by using an ROI identification technology, so that the determination efficiency of the part of interest of the user in the screen image can be greatly improved, the image coding can be realized in an efficient manner, and the improvement of the image coding efficiency can be realized.

The image coding method provided by the embodiment of the application can be applied to electronic equipment, for example, the image coding method can be specifically completed by an encoder in the electronic equipment, and taking a display layer as a window as an example, fig. 7 shows an electronic equipment component architecture for realizing the image coding method provided by the embodiment of the application, and as shown in fig. 7, the component architecture can comprise a window system, a window manager and an encoder;

The window system can open a plurality of windows, such as window 1 and window 2 in the example of fig. 7, wherein window 1 is a top-level window and covers a part of the area of the lower-level window 2;

the window manager can acquire information such as display level, position information (such as boundary point positions of the windows) and resolution of each window from the window system, and the window manager can transmit the information to the encoder so that the encoder can encode screen images (such as desktop images);

The encoder can acquire information such as display levels, position information, resolution and the like of each window from the window manager, and encode and compress screen images at least based on the display levels of each window; in configuring the compressed image quality block, the embodiment of the application can be realized by using quantization coefficients, such as setting lower quantization coefficients for the window displayed on the upper layer, namely configuring smaller quantization coefficients when needing higher image quality and configuring larger quantization coefficients when needing lower image quality, so as to achieve the purpose of configuring quantization coefficients of the region of interest to the user and quantization coefficients of the region not of interest to the user according to the display level of the window;

In logic implementation, an encoder can determine a display level of a window in a screen image from a window manager, determine a top-level window in the screen image according to the display level, and determine a first display area of interest to a user in the screen image based at least on the top-level window, thereby setting a compressed image quality of the first display area higher than a compressed image quality of a second display area in the screen image (such as a quantization coefficient of the first display area and lower than a quantization coefficient of the second display area), wherein the second display area is a display area outside the first display area in the screen image, and further encode the screen image according to the compressed image quality of the first display area and the compressed image quality of the second display area to obtain an encoded image;

Through the processing of the encoder, if the window of interest to the user is only window 1 in the example of fig. 7, the effect is achieved that the image quality of window 1 is guaranteed, while the image quality of window 2 is even lower, without great influence on the user, so that the image encoding can be achieved while the image quality and the image compression rate are both taken into consideration.

In a more specific description, when determining a first display area of interest to a user in a screen image, an encoder can rely on a display level obtained from a window manager and combine position information and resolution information of each window, specifically, in combination with the example of fig. 7, if the window of interest to the user is only window 1, the range enclosed by 4 boundary point positions (A1, B1, C1, D1) of the window 1 is the first display area of interest to the user, and the area outside the range enclosed by 4 boundary point positions (A1, B1, C1, D1) of the window 1 can be regarded as a second display area of no interest to the user, and the encoder configures different compressed image qualities for the first display area and the second display area respectively, so that the configuration of higher compressed image quality for the first display area and the configuration of lower compressed image quality for the second display area can improve the compression efficiency under the condition of basically no sense of the user.

In further explanation, the embodiment of the present application may add a window activity detection function and a window validity judgment function to the encoder, so as to make the detection of the first display area of interest to the user in the screen image more accurate, and the following description is made with respect to possible improvement scenarios:

1. Based on the window activity detection function, if the window 2 is detected to display video or animation, that is, the frame of the window 2 has a larger change in the time domain, and based on the window validity judging function, if the area proportion of the detection window 1 which is not blocked is not less than a first proportion (for example, 60%), the placement of the window 1 and the window 2 may be deliberate by a user, at this time, in order to reduce misjudgment of the first display area, the encoder may expand the range of the first display area interested by the user on the basis of the top-level window 1, so that the first display area includes the display areas of the window 1 and the window 2, for example, the first display area is set as a range enclosed by four boundary points (A2, B2, C2, D2) of the window 2 and four boundary points (A1, B1, C1, D1) of the window 1;

2. Based on the window effectiveness judging function, if the proportion of the area of the window 1 to the screen is judged to be smaller than the second proportion (such as 20%), and the area of the window 2 is not smaller than the area of the window 1 of the first multiple (such as 2 times, etc.), the area of the window 1 of the top layer is smaller, so that the user generally cannot focus on the window 1 after scaling the window 1 of the top layer from the perspective of view, and therefore, in order to reduce misjudgment of the first display area, the encoder can expand the range of the first display area interested by the user on the basis of the window 1 of the top layer, so that the first display area comprises the display areas of the window 1 and the window 2, such as the area surrounded by four boundary points (A2, B2, C2, D2) of the window 2 and four boundary points (A1, B1, C1, D1) of the window 1;

3. Based on the window validity judging function, if it is judged that the window 1 is not at the center position of the screen image, for example, the window 1 is at the corner positions of the upper left, lower left, upper right, lower right, etc. of the screen, the user generally does not focus on these corner positions from the viewpoint of viewing, so in order to reduce erroneous judgment of the first display area, the encoder may expand the range of the first display area of interest to the user on the basis of the top-level window 1 so that the first display area includes the display areas of the window 1 and the window 2, such as setting the first display area as the range enclosed by the four boundary points (A2, B2, C2, D2) of the window 2 and the four boundary points (A1, B1, C1, D1) of the window 1.

The foregoing describes the encoder further determining the target display layer of interest to the user based on the top display layer to determine the possible ways of determining the first display area of interest to the user, these ways being merely alternative ways of further perfecting the first display area of interest to the user based on the top display layer, and other possible alternatives are fully available to the encoder based on the foregoing description and are not further developed herein.

After the encoder finishes encoding the image, the encoded image may be further processed, for example, the encoded image may be transmitted to another device, decoded by another device, and the decoded image may be displayed.

In the embodiment of the application, the encoder can identify the display level of the window through the window manager, so that the first display area which is interested by the user in the screen image is determined at least according to the display level of the window, and further different compression image quality is set for the first display area which is interested by the user and the second display area which is not interested by the user, so that the encoding work of the screen image is realized; compared with an image coding scheme based on ROI identification, the method and the device can determine the first display region of interest of a user based on the display level of the window in the screen image in a multi-window display environment, and the region of interest of the user in the screen image is not required to be identified by the ROI identification technology, so that the determination efficiency of the region of interest of the user in the screen image can be greatly improved, image coding can be realized in an efficient mode, and by setting different compression configurations of the first display region and the second display region, the method and the device can reduce the overall data volume of the coded image on the premise of guaranteeing the image quality of the first display region in the screen image, improve the image compression rate, and improve the image coding efficiency under the condition of considering the image quality and the image compression efficiency.

For the image encoding method provided by the embodiment of the application, the embodiment of the application also provides a corresponding image decoding method which can be applied to electronic equipment, taking a screen sharing service as an example, the electronic equipment applying the image decoding method can be used for example, the electronic equipment acquiring the shared screen image can be used for example, the electronic equipment applying the image decoding method can be used for example, the terminal equipment using the cloud screen service and the like, and in an optional implementation, fig. 8 shows an optional flow of the image decoding method provided by the embodiment of the application, as shown in fig. 8, the flow can comprise:

step S300, obtaining an encoded image corresponding to a screen image, wherein the compressed image quality of a first display area interested by a user in the encoded image is higher than that of a second display area, the second display area is a display area outside the first display area, and the first display area at least comprises a top display layer in the screen image.

And step S310, decoding the coded image.

The image encoding process of the embodiment of the application is realized based on the compressed image quality of the first display area and is higher than the compressed image quality of the second display area, so that the image quality of the first display area in the decoded image is better than the image quality of the second display area after the image is decoded, namely the definition and the image quality of the first display area which are interested by a user can be ensured. Since image decoding and image encoding are reciprocal processes, embodiments of the present application are not further developed herein.

Further, as a possible use case, in the image encoding process, the display layer in the screen image according to the embodiment of the present application may be a plurality of windows opened by different applications (application programs), and/or at least one window opened by one application, where possible use cases are as follows:

the user takes the desktop as the background, opens windows of a plurality of applications, and one application possibly opens at least one window, so that the windows are displayed in a layered and overlapped mode by taking the desktop as the background, and when the user needs to code the desktop image, the user can complete the encoding work of the desktop image by using the image encoding method provided by the embodiment of the application;

when a user opens a plurality of windows in an application, the windows may be displayed in a layered and overlapped manner on an application interface of the application, and when the user needs to code an application interface image, the user can complete the coding work of the application interface image by using the image coding method provided by the embodiment of the application.

From the aspect of service deployment, the image coding method provided by the embodiment of the application can be deployed in a screen sharing service (such as a desktop sharing service, a remote desktop service and the like), a cloud screen service (such as a cloud desktop service and the like) and the like, and the image coding scheme provided by the embodiment of the application is described by taking the desktop sharing service as an example.

As shown in fig. 9, when the terminal a needs to share the desktop image with the terminal B through a desktop sharing service (such as a network that provides the desktop sharing service), the terminal a needs to encode the desktop image of each frame to form a video encoded image, so that the video encoded image is transmitted to the terminal B, so that the terminal B can display the desktop image shared by the terminal a;

For any frame of desktop image to be encoded of the terminal A, an encoder of the terminal A can determine a display level of a display layer (such as a window) in the desktop, determine a top display layer in the desktop, further analyze whether a target display layer interesting to a user exists in the display layer of the desktop except the top display layer, if so, take a display area corresponding to the top display layer and the analyzed target display layer as a first display area interesting to the user, and if not, directly take the display area corresponding to the top display layer as the first display area interesting to the user;

In the processing procedure of the terminal A, the terminal A can reduce the whole data volume after the desktop image coding and improve the image coding efficiency on the premise of guaranteeing the image quality of the first display area interested by the user in the desktop, has the advantage of saving the transmission bandwidth of the video coding image, and can realize reliable and efficient desktop sharing service.

The foregoing describes several embodiments of the present application, and the various alternatives presented by the various embodiments may be combined, cross-referenced, with each other without conflict, extending beyond what is possible embodiments, all of which are considered to be embodiments of the present application disclosed and disclosed.

The image coding device provided by the embodiment of the application is introduced below, the image coding device described below can be regarded as an electronic device, and the content of the image coding device described below can be correspondingly referred to the content of the image coding method described above for realizing the functional module required for the image coding method provided by the embodiment of the application.

In an alternative implementation, fig. 10 shows an alternative block diagram of an image encoding apparatus provided by an embodiment of the present application, and referring to fig. 10, the image encoding apparatus may include:

a display level determining module 100, configured to determine a display level of a display layer in a screen image;

A region of interest determining module 110, configured to determine a top display layer in the screen image according to the display hierarchy, and determine a first display region of interest to a user in the screen image based at least on the top display layer;

The compression setting module 120 is configured to set a compressed image quality of the first display area to be higher than a compressed image quality of a second display area in the screen image, where the second display area is a display area other than the first display area in the screen image;

And the encoding module 130 is configured to encode the screen image according to the compressed image quality of the first display area and the compressed image quality of the second display area, so as to obtain an encoded image.

Optionally, the display layers in the screen image are multiple, and the interest area determining module 110 is configured to determine, based on at least the top display layer, a first display area of interest to the user in the screen image, where the first display area includes:

judging whether a target display layer which is interested by a user exists in the screen image or not except the top display layer;

and if the target display layer which is interested by the user exists in the screen image, determining the top display layer and the display area corresponding to the target display layer as the first display area.

Optionally, the region of interest determining module 110 is configured to determine whether, in addition to the top-layer display layer, there is a target display layer of interest to the user in the screen image, including:

judging whether a user is interested in the next display layer of the top display layer, wherein the next display layer is a display layer with a display level lower than that of the top display layer by one level;

Correspondingly, the region of interest determining module 110 is configured to determine, if there is a target display layer of interest to the user in the screen image, the top display layer and a display region corresponding to the target display layer as the first display region, where the determining includes:

and if the user is interested in the next display layer, determining the top display layer and the display area corresponding to the next display layer as the first display area.

Optionally, in an aspect, the region of interest determining module 110 is configured to determine whether the user is interested in a next display layer of the top display layer includes:

and judging whether the user is interested in the next display layer or not at least according to the display content of the next display layer.

Optionally, the region of interest determining module 110 is configured to determine, at least according to the display content of the next display layer, whether the user is interested in the next display layer includes:

And judging whether the next display layer displays dynamic content, and whether the area proportion of the next display layer which is not blocked is not smaller than a set first proportion, wherein if the next display layer is judged to display dynamic content, and the area proportion of the next display layer which is not blocked is not smaller than the set first proportion, the user is interested in the next display layer.

Optionally, in another aspect, the region of interest determining module 110 is configured to determine whether the user is interested in a next display layer of the top display layer includes:

And judging whether the user is interested in the next display layer or not according to the area of the top display layer and the area relation between the top display layer and the next display layer.

Optionally, the region of interest determining module 110 is configured to determine whether the user is interested in the next display layer of the top display layer according to the area of the top display layer and the area relationship between the top display layer and the next display layer, including:

And judging whether the area proportion of the top display layer occupying the screen image is not more than a set second proportion and whether the area of the next display layer is at least a first multiple of the area of the top display layer, wherein if the area proportion of the top display layer occupying the screen image is not more than the set second proportion and the area of the next display layer is at least the first multiple of the area of the top display layer, the user is interested in the next display layer.

Optionally, in still another aspect, the region of interest determining module 110 is configured to determine whether the user is interested in a next display layer of the top display layer includes:

and judging whether the top display layer is at the main body position of the screen image, wherein if the top display layer is judged not to be at the main body position of the screen image, the user is interested in the next display layer.

Optionally, fig. 11 shows a further alternative block diagram of an image encoding apparatus according to an embodiment of the present application, and in combination with fig. 10 and fig. 11, the apparatus may further include:

a position determining module 140, configured to determine position information of each display layer in the screen image, where the position information of the display layer includes a boundary point position of the display layer;

correspondingly, the region of interest determining module 110 is configured to determine, as the first display region, a display region corresponding to the top display layer and the next display layer, where the determining includes:

And determining the display area corresponding to the boundary point position of the top display layer and the boundary point position of the next display layer as the first display area.

judging whether a target display layer added with an interesting label exists in the screen image except the top display layer;

and if the target display layer added with the interesting label exists in the screen image, determining the top display layer and the display area corresponding to the target display layer added with the interesting label as the first display area.

Optionally, the display layers in the screen image are one or more, and in a possible alternative implementation, the region of interest determining module 110 is configured to determine, based at least on the top display layer, a first display region of interest to the user in the screen image, including:

and determining the display area corresponding to the top display layer as the first display area.

Optionally, when the display layers in the screen image are one, the second display area comprises a background area except for the display layers in the screen image, and when the display layers in the screen image are multiple, the second display area comprises display areas corresponding to other display layers except for the top display layer, and the background area except for the display layers in the screen image.

Optionally, if multiple display layers exist in the screen image, the multiple display layers in the screen image include the top display layer, a target display layer which is interested by a user except the top display layer, and a non-target display layer which is not interested by the user, the first display region includes a display region corresponding to the top display layer and a display region corresponding to the target display layer, and the second display region includes a display region corresponding to the non-target display layer and a background region except the display layer in the screen image.

Optionally, after determining the first display area and the second display area in the screen image, the compression setting module 120 is configured to set a compressed image quality of the first display area to be higher than a compressed image quality of the second display area in the screen image, and may include:

Setting a quantization coefficient of the first display area lower than a quantization coefficient of a second display area in the screen image, wherein the quantization coefficient and the quality of the compressed image are in a negative correlation;

Accordingly, the encoding module 130 is configured to encode the screen image according to the compressed image quality of the first display area and the compressed image quality of the second display area, and the obtaining the encoded image includes:

And encoding the screen image according to the quantized coefficients of the first display area and the quantized coefficients of the second display area to obtain an encoded image.

Optionally, the compression setting module 120 is configured to set quantization coefficients of the first display area, and quantization coefficients lower than quantization coefficients of the second display area in the screen image include:

And setting a first quantized coefficient corresponding to the ROI for the first display region, and setting a second quantized coefficient corresponding to the non-ROI for the second display region, wherein the first quantized coefficient is lower than the second quantized coefficient.

Optionally, if the plurality of display layers in the screen image include the top display layer, a target display layer that is of interest to the user except for the top display layer, and a non-target display layer that is not of interest to the user, in an optional implementation, the compression setting module 120 is configured to set quantization coefficients of the first display area, where the quantization coefficients lower than quantization coefficients of the second display area in the screen image include:

Setting the quantization coefficients of the top display layer and the target display layer to be the same and lower than those of the non-target display layer, wherein when the number of the non-target display layers is multiple, the quantization coefficients of the non-target display layer are increased along with the reduction of the display level, and setting the quantization coefficients of the background area to be not lower than those of the non-target display layer of the lowest display level.

Optionally, in one use case, the display layer in the screen image comprises a plurality of windows opened by different applications, wherein one application opens at least one window, the plurality of windows are displayed in a layered and overlapped mode by taking a desktop as a background, and in another use case, the display layer in the screen image comprises a plurality of windows opened in one application, and the plurality of windows are displayed in a layered and overlapped mode on an application interface of the application.

Alternatively, as an application, the electronic device for performing image coding may be a terminal device, where the screen image is a local screen image, further, fig. 12 shows another optional block diagram of the image coding apparatus provided by the embodiment of the present application, and in conjunction with fig. 10 and fig. 12, the apparatus may further include:

A screen sharing module 150, configured to share the encoded image to other terminal devices participating in a screen sharing service;

Accordingly, in an alternative implementation, the local screen image is a local desktop image, and the screen sharing service includes a desktop sharing service or a remote desktop service.

Alternatively, as another application, the electronic device for performing image encoding may be a cloud service device for providing a cloud screen service, where the screen image is a cloud screen image, further, fig. 13 shows a further optional block diagram of an image encoding apparatus provided by an embodiment of the present application, and in conjunction with fig. 10 and fig. 13, the apparatus may further include:

A cloud screen transmission module 160 for transmitting the encoded image to a terminal device using a cloud screen service;

accordingly, in an alternative implementation, the cloud screen image includes a cloud desktop image, and the cloud screen service includes a cloud desktop service.

In an alternative implementation, the functions of the image encoding apparatus provided in the embodiment of the present application may be implemented in an encoder of an electronic device (possibly a terminal device or possibly a cloud service device), and specifically, in conjunction with fig. 7, the image encoding apparatus may include an encoder;

The encoder is used for determining a display level of a window in a screen image, determining a top-layer window in the screen image according to the display level, determining a first display area which is interested by a user in the screen image at least based on the top-layer window, setting the compressed image quality of the first display area to be higher than the compressed image quality of a second display area in the screen image, wherein the second display area is a display area outside the first display area in the screen image, and encoding the screen image according to the compressed image quality of the first display area and the compressed image quality of the second display area to obtain an encoded image.

Optionally, the image encoding apparatus may further include:

A window system for opening a window;

and a window manager for acquiring at least a display level of each window from the window system, wherein the window manager transmits at least the acquired display level to the encoder.

The image coding device provided by the embodiment of the application can improve the image coding efficiency under the condition of considering both the image quality and the image compression efficiency, and correspondingly, the electronic equipment (possibly the terminal equipment and the cloud service equipment) for executing the image coding can realize the image coding method provided by the embodiment of the application by loading the image coding device.

As an alternative implementation, fig. 14 shows an alternative block diagram of an electronic device provided by an embodiment of the application, comprising at least one processor 1, at least one communication interface 2, at least one memory 3 and at least one communication bus 4;

In the embodiment of the application, the number of the processor 1, the communication interface 2, the memory 3 and the communication bus 4 is at least one, and the processor 1, the communication interface 2 and the memory 3 complete the communication with each other through the communication bus 4;

alternatively, the communication interface 2 may be an interface of a communication module for performing network communication;

Alternatively, the processor 1 may be a CPU (central processing unit), GPU (Graphics Processing Unit, graphics processor), NPU (embedded neural network processor), FPGA (Field Programmable GATE ARRAY ), TPU (tensor processing unit), AI chip, application-specific ASIC (Application SPECIFIC INTEGRATED Circuit), or one or more integrated circuits configured to implement embodiments of the present application, or the like.

The memory 3 may comprise a high-speed RAM memory or may further comprise a non-volatile memory, such as at least one disk memory.

Wherein the memory 3 stores one or more computer-executable instructions that the processor 1 invokes to perform the image encoding method provided by the embodiments of the present application.

The embodiment of the application also provides a storage medium which stores one or more computer executable instructions which when executed implement the image encoding method provided by the embodiment of the application.

The embodiment of the application also provides an image decoding device which can be regarded as an electronic device, and is a functional module required to be set for realizing the image decoding method provided by the embodiment of the application, and the content of the image decoding device described below can be correspondingly referred to with the content described above.

In an alternative implementation, fig. 15 shows an alternative block diagram of an image decoding apparatus provided by an embodiment of the present application, and referring to fig. 15, the image decoding apparatus may include:

The encoded image obtaining module 200 is configured to obtain an encoded image corresponding to a screen image, where the compressed image quality of a first display area interested by a user in the encoded image is higher than the compressed image quality of a second display area, where the second display area is a display area other than the first display area, and the first display area at least includes a top display layer in the screen image;

A decoding module 210, configured to decode the encoded image.

In an optional implementation, the function of the image decoding device provided by the embodiment of the application may be implemented in a decoder of an electronic device (possibly a terminal device or possibly a cloud service device), and specifically the image decoding device may include a decoder, where the decoder is configured to obtain an encoded image corresponding to a screen image, and the compressed image quality of a first display area interested by a user in the encoded image is higher than the compressed image quality of a second display area, where the second display area is a display area other than the first display area, and the first display area includes at least a top layer window in the screen image, and decode the encoded image.

The decoding device provided by the embodiment of the application has the advantage that the image quality of the first display area in the decoded image is better than that of the second display area, namely the definition and the image quality of the first display area interested by a user can be ensured. Accordingly, the electronic device (possibly the terminal device or the cloud service device) for performing image decoding may implement the image decoding method provided by the embodiment of the present application by loading the image decoding device described above. Alternatively, a block diagram of an electronic device for performing image decoding may be combined with the one described with reference to fig. 14, and include at least one memory storing one or more computer-executable instructions and at least one processor invoking the one or more computer-executable instructions to perform the image decoding method provided by the embodiments of the present application.

The embodiment of the application also provides a storage medium which stores one or more computer executable instructions which when executed implement the image decoding method provided by the embodiment of the application.

Although the embodiments of the present application are disclosed above, the present application is not limited thereto. Various changes and modifications may be made by one skilled in the art without departing from the spirit and scope of the application, and the scope of the application should be assessed accordingly to that of the appended claims.

Claims

1. An image encoding method, wherein applied to an electronic device, the method comprising:

Determining a display level of a display layer in the screen image;

Encoding the screen image according to the compressed image quality of the first display area and the compressed image quality of the second display area to obtain an encoded image;

The method comprises the steps of determining whether a target display layer which is interested by a user exists in a screen image or not except the top display layer, and determining the top display layer and a display area corresponding to the target display layer as the first display area if the target display layer which is interested by the user exists in the screen image.

2. The image encoding method according to claim 1, wherein the determining whether there is a target display layer of interest to the user in the screen image in addition to the top-layer display layer includes:

If the target display layer of interest to the user also exists in the screen image, determining the top display layer and the display area corresponding to the target display layer as the first display area includes:

3. The image encoding method of claim 2, wherein the determining whether the user is interested in a next display layer of the top display layer comprises:

4. The image encoding method of claim 3, wherein the determining whether the user is interested in the next display layer based at least on the display content of the next display layer comprises:

And judging whether the next display layer displays dynamic content, and whether the area proportion of the next display layer which is not blocked is not smaller than a set first proportion, wherein if the next display layer is judged to display dynamic content, and the area proportion of the next display layer which is not blocked is not smaller than the set first proportion, a user is interested in the next display layer.

5. The image encoding method of claim 2, wherein the determining whether the user is interested in a next display layer of the top display layer comprises:

6. The image encoding method according to claim 5, wherein the determining whether the user is interested in the next display layer according to the area of the top display layer and the area relationship between the top display layer and the next display layer comprises:

7. The image encoding method of claim 2, wherein the determining whether the user is interested in a next display layer of the top display layer comprises:

8. The image encoding method according to claim 2, further comprising:

determining position information of each display layer in a screen image, wherein the position information of the display layer comprises boundary point positions of the display layer;

The determining the display areas corresponding to the top display layer and the next display layer as the first display area includes:

9. The image encoding method according to claim 1, wherein the determining whether there is a target display layer of interest to the user in the screen image in addition to the top-layer display layer includes:

10. The image encoding method of claim 1, wherein the display layers in the screen image are one or more, and wherein the determining the first display area of interest to the user in the screen image based at least on the top display layer comprises:

11. The image encoding method according to claim 10, wherein the second display area includes a background area except for a display layer in the screen image when the display layer in the screen image is one, and includes a display area corresponding to other display layers except for the top display layer when the display layer in the screen image is a plurality of display layers, and a background area except for the display layer in the screen image.

12. The image encoding method of claim 1, wherein the plurality of display layers in the screen image include the top display layer, target display layers of interest to a user except the top display layer, and non-target display layers of no interest to the user, the first display region includes a display region corresponding to the top display layer and a display region corresponding to the target display layer, and the second display region includes a display region corresponding to the non-target display layer and a background region except the display layer in the screen image.

13. The image encoding method according to any one of claims 1 to 12, wherein said setting the compressed image quality of the first display area higher than the compressed image quality of the second display area in the screen image includes:

the encoding the screen image according to the compressed image quality of the first display area and the compressed image quality of the second display area, and the obtaining the encoded image includes:

14. The image encoding method of claim 13, wherein the setting the quantization coefficients of the first display region to be lower than the quantization coefficients of the second display region in the screen image comprises:

and setting a first quantized coefficient corresponding to a region of interest (ROI) for the first display region, and setting a second quantized coefficient corresponding to a non-ROI for the second display region, wherein the first quantized coefficient is lower than the second quantized coefficient.

15. The image encoding method of claim 13, wherein the plurality of display layers in the screen image comprises:

the top layer display layer, except for the top layer display layer, a target display layer which is interested by a user and a non-target display layer which is not interested by the user;

the setting the quantization coefficient of the first display area, which is lower than the quantization coefficient of the second display area in the screen image, includes:

setting the quantization coefficients of the top display layer and the target display layer to be the same and lower than those of the non-target display layer, wherein when the number of the non-target display layers is multiple, the quantization coefficients of the non-target display layer are increased along with the reduction of the display level, and setting the quantization coefficient of the background area to be not lower than that of the non-target display layer of the lowest display level.

16. The image encoding method according to claim 1, wherein the display layer in the screen image comprises a plurality of windows opened by different applications, wherein one application opens at least one window, and the plurality of windows are displayed in a layered and overlapped manner by taking a desktop as a background;

or the display layer in the screen image comprises a plurality of windows which are opened in one application and are displayed in a layered and overlapped mode on an application interface of the application.

17. The image encoding method according to claim 1, wherein the electronic device is a terminal device and the screen image is a local screen image.

18. The image encoding method of claim 17, wherein the method further comprises:

and sharing the coded image to other terminal devices participating in the screen sharing service.

19. The image encoding method of claim 18, wherein the local screen image is a local desktop image;

The screen sharing service includes a desktop sharing service or a remote desktop service.

20. The image encoding method according to claim 1, wherein the electronic device is a cloud service device that provides a cloud screen service, the screen image is a cloud screen image, the method further comprising:

and transmitting the encoded image to a terminal device using a cloud screen service.

21. The image encoding method of claim 20, wherein the cloud screen image comprises a cloud desktop image and the cloud screen service comprises a cloud desktop service.

22. An image decoding method, wherein applied to an electronic device, the method comprising:

Acquiring a plurality of coded images corresponding to a screen image, wherein the number of display layers in the screen image is greater than that of a first display area interested by a user in the coded images, the compressed image quality is higher than that of a second display area, the second display area is a display area except for the first display area, the first display area at least comprises a top display layer and a target display layer in the screen image, and the target display layer is a display layer which exists in the screen image and is interested by the user except for the top display layer;

decoding the encoded image.

23. An image encoding apparatus, comprising:

the encoding module is used for encoding the screen image according to the compressed image quality of the first display area and the compressed image quality of the second display area to obtain an encoded image;

The interest area determining module is further used for judging whether a target display layer which is interested by a user exists in the screen image or not besides the top display layer, and determining the top display layer and the display area corresponding to the target display layer as the first display area if the target display layer which is interested by the user exists in the screen image.

24. An image encoding apparatus includes an encoder; the encoder is used for:

the method comprises the steps of determining a display level of a window in a screen image, determining a top-level window in the screen image according to the display level, determining a first display area of interest to a user in the screen image at least based on the top-level window, setting the compressed image quality of the first display area to be higher than the compressed image quality of a second display area in the screen image, wherein the second display area is a display area outside the first display area in the screen image, and encoding the screen image according to the compressed image quality of the first display area and the compressed image quality of the second display area to obtain an encoded image;

The method comprises the steps of determining whether a target display layer which is interested by a user exists in a screen image or not except the top-layer window, and determining the top-layer window and a display area corresponding to the target display layer as the first display area if the target display layer which is interested by the user exists in the screen image.

25. The image encoding apparatus according to claim 24, further comprising a window system for opening a window;

26. An image decoding apparatus, comprising:

The coding image acquisition module is used for acquiring coding images corresponding to screen images, wherein a plurality of display layers are arranged in the screen images, the compression image quality of a first display area which is interested by a user in the coding images is higher than that of a second display area which is a display area except the first display area, the first display area at least comprises a top display layer and a target display layer which are arranged in the screen images, and the target display layer is a display layer which is interested by the user except the top display layer;

And the decoding module is used for decoding the coded image.

27. An electronic device comprising at least one memory storing one or more computer-executable instructions and at least one processor invoking the one or more computer-executable instructions to perform the image encoding method of any of claims 1-21 or the image decoding method of claim 22.

28. A storage medium storing one or more computer-executable instructions which, when executed, implement the image encoding method of any one of claims 1-21 or the image decoding method of claim 22.