- 8399 Views
- 7 replies
- 2 kudos
Installing linux packages on cluster
Hey everyone! We have a need to utilize libreoffice in one of our automated tasks via a notebook. I have tried to install via a init script that I attach to the cluster, but sometimes the program gets installed and sometimes it doesn't. For obviou...
- 8399 Views
- 7 replies
- 2 kudos
- 2 kudos
It only works in the driver, when I try to use the whole cluster the nodes can't access the command.
- 2 kudos
- 7612 Views
- 5 replies
- 1 kudos
Resolved! Query text truncated for queries longer than 153,596 characters
Hi, When using the `query_history.list` function of the python SDK workspace client the queries that have more than 153,596 characters are truncated.I could not find anywhere in the documentation this limit so I wanted to know if this is documented s...
- 7612 Views
- 5 replies
- 1 kudos
- 1 kudos
Hi @mangosta , I did some testing internally and was able to replicate the behavior you described. The query text limit is a limitation not of the SDK or the API, but rather of the backing system table `system.query.history`. More information on thi...
- 1 kudos
- 1295 Views
- 1 replies
- 2 kudos
Can't create tables in unity catalog
I am facing the problem that is very similar to https://community.databricks.com/t5/data-engineering/can-t-create-table-in-unity-catalog/td-p/93414. However, there is no answer there so I create a new post.I created my workspace with terraform, with ...
- 1295 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @Kit , Greetings! Can you try to create a table with the following SQL command: CREATE OR REPLACE TABLE table_name (). Also can you please which cluster you are using here? Make sure you are using the UC-supported cluster. Leave a like if this he...
- 2 kudos
- 1190 Views
- 1 replies
- 3 kudos
Resolved! Not able to read data from Delta External table in catalog
spark.conf.set( "fs.azure.account.key.sa02flexflowinpp01prod.dfs.core.windows.net", dbutils.secrets.get(scope="OpenScope", key="sa02StorageAccessKey") I created an external table using this configuration I am able to query the data only when I ...
- 1190 Views
- 1 replies
- 3 kudos
- 3 kudos
Hi @bohemiaRDX , Greetings! Generally, this error would occur if the path is not added as an external location with storage credentials. Here the cluster could be trying to access the storage which doesn’t have UC storage credentials set nor any non...
- 3 kudos
- 2215 Views
- 1 replies
- 0 kudos
Scanning Unity Catalog Schema and sample data
Hey Guyz, We need to scan the complete schema present at unity catalog to an external user or group of users without onboarding them on to our platform. Is there a way we can expose this details to them. Additionally, can we expose the sample dataset...
- 2215 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @RohitKumar7 , Greetings! Looking at your request, i would like to confirm you that it would be possible to use the Delta sharing feature. Delta sharing feature lets you share data and AI assets with users outside your organization, whether or not...
- 0 kudos
- 2571 Views
- 2 replies
- 2 kudos
Unity Catalog and environment set up
We are implementing the Databricks medallion architecture (bronze, silver, gold). We have three different environments/workspaces in Databricks: Dev, Test and Prod. Each catalog in Unity Catalog points to a specific place in the Azure Data Lake. It t...
- 2571 Views
- 2 replies
- 2 kudos
- 2 kudos
Thanks, Yes, that is indeed an option. The issue there is that we loose some flexibility in the sense that we cant define other sub-schemas to gold, silver, bronze as it would then be of the form prod.gold.<table-name>. instead of gold_dev.<schema-na...
- 2 kudos
- 7037 Views
- 3 replies
- 2 kudos
Setup learning environment failed: Configuration dbacademy.library.version is not available.
Using 1.3 Getting Started with the Databricks Platform Lab. to self learning. When I run DE 2.1 to setup environment, got following error:Configuration dbacademy.library.version is not available.Following is the code in the common setup.specified_ve...
- 7037 Views
- 3 replies
- 2 kudos
- 2 kudos
HiI resolved adding some instructions in the _common notebook, you can find inside the folder IncludesPut these at the beginning%pip install git+https://github.com/databricks-academy/dbacademy@v3.0.70%python dbutils.library.restartPython() After this...
- 2 kudos
- 1025 Views
- 1 replies
- 0 kudos
Geometry Point and WKB based on latitude and longitude
HiWhat is the best method to generate Geometry Point and WKB based on latitude and longitude stored in a Databricks table? Thanks,Sam
- 1025 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi,Spark has function to work with geospatial data, for instance ST_GeomFromWKB. You can use it to convert it human readable form. You can also create UDFs if something is missing. In my project I stored latitude and longitude as separate columns.
- 0 kudos
- 1466 Views
- 1 replies
- 0 kudos
Rename columns in Delta Live Tables
I want to explore the option of renaming a column in the SILVER layer of a DLT pipeline. Requesting suggestions.
- 1466 Views
- 1 replies
- 0 kudos
- 0 kudos
Full reload will rename column if it's caused by a column rename in a source file.
- 0 kudos
- 4062 Views
- 2 replies
- 0 kudos
Changes to deletion behavior of Materialized View and Streaming Tables defined by Delta Live Table
Hi,Sometime ago, I got a message that there will be a change (starting from 01/31/2025) in "deletion behavior of Materialized View and Streaming Tables defined by Delta Live Table", but when I remove dlt pipeline, it also removes related tables, will...
- 4062 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @MariuszK, The users will need to explicitly call DROP MATERIALIZED VIEW to delete MVs and DROP TABLE to delete STs, when deleting DLT pipelines. https://home.databricks.com/account-alert-deletion-behavior-change-for-materialized-view-and-streamin...
- 0 kudos
- 2166 Views
- 3 replies
- 2 kudos
Resolved! Instance Pool Usage
We have instance pools setup with a maximum capacity and are looking at ways to monitor the usage to help with our capacity planning.I have been using the system tables to track how many nodes are being used within a pool at a point in time but it ap...
- 2166 Views
- 3 replies
- 2 kudos
- 1983 Views
- 1 replies
- 2 kudos
Tableau analytics integration with databricks delta lake
Hii there!!Currently, we are exploring options for reporting on Salesforce. We extract data from salesforce via databricks and store it in delta lake.Is there a connector by which data can be pulled from databricks into Tableau/CRM analytics??I know ...
- 1983 Views
- 1 replies
- 2 kudos
- 2 kudos
Hello @ruby Were you able to resolve this? I know it's been a while, but I believe we now have multiple ways to connect Tableau and Databricks.1. Use the Native Databricks Connector for TableauTableau has a built-in Databricks connector, making it ea...
- 2 kudos
- 2874 Views
- 5 replies
- 0 kudos
Delta sharing for external table to external users who has no access to external storage?
We used delta sharing (authentication type: token) to generate the config.share file and share with external users not from our organisation, the users faced the "FileNotFoundError" error while using python "delta_sharing.load_as_pandas" method to re...
- 2874 Views
- 5 replies
- 0 kudos
- 0 kudos
Hello @tonykun_sg,It looks like ADLS Gen2 might be restricting access to the data through an ACL, which is why Databricks allows access but the underlying files remain protected. Could you check with your team to temporarily enable access for testing...
- 0 kudos
- 6692 Views
- 8 replies
- 6 kudos
Resolved! Workflow SQL Task Query Showing Empty
I am trying to create a SQL task in Workflows. I have my query which executes successfully in the SQL editor, and it is saved in a repo.However, when I try to execute the task, the below error shows.Query text can not be empty: BAD_REQUEST: Query tex...
- 6692 Views
- 8 replies
- 6 kudos
- 6 kudos
It ended up being that the query wasn't actually saved. Once I manually clicked save, the query preview showed and the task ran successfully. I'm really surprised that was the reason. I had moved the query around to different folders and closed and r...
- 6 kudos
- 1377 Views
- 2 replies
- 0 kudos
my query works with All-purpose cluster but return NULL with SQL Warehouse
Hi,(1) On SQL warehouse, I created a table in unity catalog from data source file vw_businessmetrics_1000.json in ADLS blob.USE CATALOG `upreport`;USE SCHEMA `test_genie`;-- Create the external table from the JSON fileCREATE EXTERNAL TABLE IF NOT EXI...
- 1377 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Ayushi_Suthar Thank you very much. I tried with the Classic and Pro and it did not work.My solution is: drop the table and recreate as a delta table then loading data from json to the delta table. Now it works. Probably, the SQL warehouse only su...
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 572 | |
| 349 | |
| 287 |
