- 2788 Views
- 3 replies
- 1 kudos
Google BigQuery Foreign Catalog - Incorrect Data Format
I've tested a foreign catalog connected to a google bigquery project.The connection was ok and I was able to see my datasets and tablesThe problem: for columns with regular data types the data format is perfect but the columns with type record and re...
- 2788 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @RobsonNLPT, This is a limitation, the data conversion issue you are facing is expected behavior due to the current data type mappings supported by the Lakehouse Federation platform. Unfortunately, this means that the JSON format you see in Google...
- 1 kudos
- 1579 Views
- 2 replies
- 2 kudos
Parsing Japanese characters in Spark & Databricks
I'm trying to read the data which has Japanese headers, might as well have Japanese data. Currently when I say header is True, I see all jumbled characters. Can any one help how can I parse these Japanese characters correctly?
- 1579 Views
- 2 replies
- 2 kudos
- 2 kudos
Thank you, @Avinash_Narala I definitely used the encoding options to parse the data again but this time I used an encoding called `SHIFT_JIS` to solve the problem. Appreciate the quick response.!
- 2 kudos
- 12300 Views
- 6 replies
- 5 kudos
Resolved! exception when using java SQL client
Hi,I try to use java sql. i can see that the query on databricks is executed properly.However, on my client i get exception (see below).versions:jdk: jdk-20.0.1 (tryed also with version 16, same results)https://www.oracle.com/il-en/java/technologies/...
- 12300 Views
- 6 replies
- 5 kudos
- 3237 Views
- 3 replies
- 0 kudos
Resolved! Init script works fine on All purpose compute but have issues with Job compute created from DLT ETL
Hi I was following Databricks tutorial from https://notebooks.databricks.com/demos/llm-rag-chatbot the old one where it had reference on how to install OCR on nodes(install poppler on the cluster) to read the pdf content.I created below init script t...
- 3237 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi Alberto_Umana,Thanks for looking into it, I got solution from databricks support assigned for my corporation.The issue was more with cluster type and not Streaming or DLT. For Streaming I was able to use Single User compute but for DLT since we ca...
- 0 kudos
- 5005 Views
- 7 replies
- 2 kudos
Resolved! error while setting up permission for job via api
Hi Guys , I am getting below error when I am trying to setup permission for the job via api. Though I am able to create a job via api. Can anyone help to identify the issue or any one has faced below error {"error_code": "INVALID_PARAMETER_VALUE","m...
- 5005 Views
- 7 replies
- 2 kudos
- 2 kudos
I get that the solution was accepted, but it is very confusing when you run the databricks command as follows:databricks clusters get-permissions my-joyous-db-cluster"access_control_list": [{"all_permissions": [{"inherited":false,"permission_level":"...
- 2 kudos
- 2673 Views
- 1 replies
- 1 kudos
Resolved! Cluster install of Python libraries versus notebook install
If a base set of libraries is installed on the cluster and pinned to a specific version, can/would this conflict with a notebook submitted to the cluster that defines a conflicting set of libraries for install?Is there a way to override the cluster p...
- 2673 Views
- 1 replies
- 1 kudos
- 1 kudos
When a base set of libraries is installed on a cluster, can indeed conflict with a notebook submitted to the cluster that defines a conflicting set of libraries for installation. This is because the libraries installed at the cluster level take prece...
- 1 kudos
- 4981 Views
- 3 replies
- 1 kudos
Can't enable CLI 2.1 on CI
Hi! this is my CI configuration, I added the databricks jobs configure --version=2.1 command but it stills showing this error, any idea of what can I be doing wrong?Error:Resetting Databricks Job with job_id 1036...WARN: Your CLI is configured to use...
- 4981 Views
- 3 replies
- 1 kudos
- 1 kudos
I got to solve this by downgrading the Databricks runtime to 13.3 and had the below commands for optimization and it worked well in my case.spark.conf.set("spark.sql.shuffle.partitions", "200")spark.conf.set("spark.sql.execution.arrow.pyspark.enabled...
- 1 kudos
- 9309 Views
- 7 replies
- 2 kudos
Installing linux packages on cluster
Hey everyone! We have a need to utilize libreoffice in one of our automated tasks via a notebook. I have tried to install via a init script that I attach to the cluster, but sometimes the program gets installed and sometimes it doesn't. For obviou...
- 9309 Views
- 7 replies
- 2 kudos
- 2 kudos
It only works in the driver, when I try to use the whole cluster the nodes can't access the command.
- 2 kudos
- 8499 Views
- 5 replies
- 1 kudos
Resolved! Query text truncated for queries longer than 153,596 characters
Hi, When using the `query_history.list` function of the python SDK workspace client the queries that have more than 153,596 characters are truncated.I could not find anywhere in the documentation this limit so I wanted to know if this is documented s...
- 8499 Views
- 5 replies
- 1 kudos
- 1 kudos
Hi @mangosta , I did some testing internally and was able to replicate the behavior you described. The query text limit is a limitation not of the SDK or the API, but rather of the backing system table `system.query.history`. More information on thi...
- 1 kudos
- 1464 Views
- 1 replies
- 2 kudos
Can't create tables in unity catalog
I am facing the problem that is very similar to https://community.databricks.com/t5/data-engineering/can-t-create-table-in-unity-catalog/td-p/93414. However, there is no answer there so I create a new post.I created my workspace with terraform, with ...
- 1464 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @Kit , Greetings! Can you try to create a table with the following SQL command: CREATE OR REPLACE TABLE table_name (). Also can you please which cluster you are using here? Make sure you are using the UC-supported cluster. Leave a like if this he...
- 2 kudos
- 1379 Views
- 1 replies
- 3 kudos
Resolved! Not able to read data from Delta External table in catalog
spark.conf.set( "fs.azure.account.key.sa02flexflowinpp01prod.dfs.core.windows.net", dbutils.secrets.get(scope="OpenScope", key="sa02StorageAccessKey") I created an external table using this configuration I am able to query the data only when I ...
- 1379 Views
- 1 replies
- 3 kudos
- 3 kudos
Hi @bohemiaRDX , Greetings! Generally, this error would occur if the path is not added as an external location with storage credentials. Here the cluster could be trying to access the storage which doesn’t have UC storage credentials set nor any non...
- 3 kudos
- 2363 Views
- 1 replies
- 0 kudos
Scanning Unity Catalog Schema and sample data
Hey Guyz, We need to scan the complete schema present at unity catalog to an external user or group of users without onboarding them on to our platform. Is there a way we can expose this details to them. Additionally, can we expose the sample dataset...
- 2363 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @RohitKumar7 , Greetings! Looking at your request, i would like to confirm you that it would be possible to use the Delta sharing feature. Delta sharing feature lets you share data and AI assets with users outside your organization, whether or not...
- 0 kudos
- 2940 Views
- 2 replies
- 2 kudos
Unity Catalog and environment set up
We are implementing the Databricks medallion architecture (bronze, silver, gold). We have three different environments/workspaces in Databricks: Dev, Test and Prod. Each catalog in Unity Catalog points to a specific place in the Azure Data Lake. It t...
- 2940 Views
- 2 replies
- 2 kudos
- 2 kudos
Thanks, Yes, that is indeed an option. The issue there is that we loose some flexibility in the sense that we cant define other sub-schemas to gold, silver, bronze as it would then be of the form prod.gold.<table-name>. instead of gold_dev.<schema-na...
- 2 kudos
- 7356 Views
- 3 replies
- 2 kudos
Setup learning environment failed: Configuration dbacademy.library.version is not available.
Using 1.3 Getting Started with the Databricks Platform Lab. to self learning. When I run DE 2.1 to setup environment, got following error:Configuration dbacademy.library.version is not available.Following is the code in the common setup.specified_ve...
- 7356 Views
- 3 replies
- 2 kudos
- 2 kudos
HiI resolved adding some instructions in the _common notebook, you can find inside the folder IncludesPut these at the beginning%pip install git+https://github.com/databricks-academy/dbacademy@v3.0.70%python dbutils.library.restartPython() After this...
- 2 kudos
- 1167 Views
- 1 replies
- 0 kudos
Geometry Point and WKB based on latitude and longitude
HiWhat is the best method to generate Geometry Point and WKB based on latitude and longitude stored in a Databricks table? Thanks,Sam
- 1167 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi,Spark has function to work with geospatial data, for instance ST_GeomFromWKB. You can use it to convert it human readable form. You can also create UDFs if something is missing. In my project I stored latitude and longitude as separate columns.
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 582 | |
| 349 | |
| 287 |
