- 9377 Views
- 5 replies
- 1 kudos
Resolved! Import python file to notebook doesn't work
I followed the documentation here under the section "Import a file into a notebook" to import a shared python file among notebooks used by delta live table. But it sometimes can find the module, sometimes not and returns me exception No module named ...
- 9377 Views
- 5 replies
- 1 kudos
- 1 kudos
Thank you so much for getting back to us @Jennifer MJ​ . It's really great of you to send in the solution. Would you be happy to mark the answer as best so other community members can find the solution quickly and easily? We really appreciate your ti...
- 1 kudos
- 9390 Views
- 2 replies
- 0 kudos
Notebook Visualisations suddenly not working
Hi all,I have a python script which runs SQL code against our Delta Live Tables and returns a pandas dataframe. I do this multiple times and then use 'display(pandas_dataframe)'. Once this displays I then create a visualization from the UI which is t...
- 9390 Views
- 2 replies
- 0 kudos
- 0 kudos
Thank you for the detailed response Kaniz, I appreciate it! I do think it may have been cache issues due to there being no spark computation when running them when the error occured.It did lead me down a train of thought.. is it possible to extract t...
- 0 kudos
- 7536 Views
- 1 replies
- 1 kudos
Difference @dlt.table and @dlt.create_table decorator
Hi!I'm currently trying to stream data from an Azure Event Hub (kafka) using DLT. The provided example (https://learn.microsoft.com/en-us/azure/databricks/delta-live-tables/event-hubs) works well.I saw in different examples the usage of two different...
- 7536 Views
- 1 replies
- 1 kudos
- 4743 Views
- 1 replies
- 1 kudos
Working with a text file that is both compressed by bz2 followed by zip in PySpark
I have downloaded Am azon reviews for sentiment analysis from here. The file is not particularly large (just over 500MB) but comes in the following formattest.ft.txt.bz2.zipSo it is a text file that is compressed by bz2 followed by zip. Now I like t...
- 4743 Views
- 1 replies
- 1 kudos
- 1 kudos
Thanks for your reply @Retired_mod On the face of it spark can handle both .bz2 and .zip . It practice it does not work with both at the same time. You end up with ineligible characters as text. I suspect it handles decompression of outer layer (in t...
- 1 kudos
- 14844 Views
- 8 replies
- 4 kudos
Resolved! Can't query delta tables, token missing required scope
Hello,I've correctly set up a stream from kinesis, but I can't read anything from my delta tableI'm actually reproducing the demo from Frank Munz: https://github.com/fmunz/delta-live-tables-notebooks/tree/main/motion-demoand I'm running the following...
- 14844 Views
- 8 replies
- 4 kudos
- 4 kudos
Hello, I also had this issue. It was because I was trying to read a DLT table with a Machine Learning Runtime. At time of writing, Machine Learning Runtimes are not compatible with shared access mode, so I ended up setting up two clusters, one MLR as...
- 4 kudos
- 1470 Views
- 0 replies
- 0 kudos
Whitelisting GraphFrame Jar files does not work for shared compute.
Hello,I'm encountering a Py4JSecurityException while using the GraphFrames jar library in a job task with shared compute. Despite following all documentation to whitelist my jar libraries in Volumes and ensuring compatibility with my Spark and Scala ...
- 1470 Views
- 0 replies
- 0 kudos
- 2515 Views
- 1 replies
- 1 kudos
Checking for availability of dynamic data masking functionality in SQL.
I am looking forward for functionality similar to snowflake which allows attaching masking to a existing column. Documents found related to masking with encryption but my use case is on the existing table. Solutions using views along with Dynamic Vie...
- 2515 Views
- 1 replies
- 1 kudos
- 1 kudos
Unity catalog provide similar feature https://docs.databricks.com/en/data-governance/unity-catalog/row-and-column-filters.html
- 1 kudos
- 4570 Views
- 2 replies
- 1 kudos
Resolved! error installing the igraph and networkD3 library
Hi!I am trying to install the igraph and networkD3 CRAN packages for use within a notebook. However, I am receiving the below installation error when attempting to do so.Could someone please assist? Thank you!* installing *source* package ‘igraph’ .....
- 4570 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi! I just figured this out myself. I'm not sure why this is suddenly occurring, since igraph has always loaded fine for me in databricks but didn't this week. I found that the following solution worked.In your notebook before installing your R libra...
- 1 kudos
- 4949 Views
- 2 replies
- 4 kudos
How to write log entries from a Delta Live Table pipeline.
From a notebook I can import the log4j logger from cs and write to a log like so:log4jLogger = sc._jvm.org.apache.log4jLOGGER = log4jLogger.LogManager.getLogger(__name__)LOGGER.info("pyspark script logger initialized")But this does not work in a Delt...
- 4949 Views
- 2 replies
- 4 kudos
- 11976 Views
- 2 replies
- 1 kudos
Reading a table from a catalog that is in a different/external workspace
I am trying to read a table that is hosted on a different workspace. We have been told to establish a connection to said workspace using a table and consume the table.Code I am using isfrom databricks import sqlconnection = sql.connect(server_hostnam...
- 11976 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi Addy Greetings! You can also use Delta sharing to share the data across multiple workspaces. Since you want to read tables from another workspace you can use databricks to databricks delta sharing. https://docs.databricks.com/en/data-sharing/read...
- 1 kudos
- 4206 Views
- 3 replies
- 0 kudos
live spark driver log analysis
In databricks, if we want to see the live log of the exuction we can able to see it from the driver log page of the cluster.But in that we can't able to search by key word instead of that we need to download every one hour log file and live logs are ...
- 4206 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @shan_chandra ,It is like we are putting our driver log into another cloud platform, But here I want to check the live log in local machine tools, is this possible?
- 0 kudos
- 4161 Views
- 5 replies
- 0 kudos
Query Serverless SQL Warehouse from Spark Submit Job
I am trying to load data from a table in SQL warehouse using spark.sql("SELECT * FROM <table>") in a spark submit job, but the job is failing with [TABLE_OR_VIEW_NOT_FOUND] The table or view . The same statement is working in notebook but not in a jo...
- 4161 Views
- 5 replies
- 0 kudos
- 0 kudos
- when you query table manually and running job - do both those actions happens in same Databricks Workspace- what is job configuration - who is job Owner or Run As Account -> do this principal/persona has access to the table ?
- 0 kudos
- 3408 Views
- 2 replies
- 0 kudos
Convert pdf's is into structured data
Is there anything on Databricks to help read PDF (payment invoices and receipts for example) and convert it to structured data?
- 3408 Views
- 2 replies
- 0 kudos
- 0 kudos
Thanks! Converting PDF format is sometimes a difficult task as not all converters provide accuracy. I want to share with you one interesting tool I recently discovered that can make your work even more efficient. I recently came across an amazing onl...
- 0 kudos
- 6525 Views
- 3 replies
- 0 kudos
Resolved! Error on Starting Databricks SQL Warehouse Serverless with Instance Profile
I have two workspaces, one in us-west-2 and the other in ap-southeast-1. I have configured the same instance profile for both workspaces. I followed the documentation to set up the instance profile for Databricks SQL Warehouse Serverless by adding th...
- 6525 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Tam , Hope you are doing well! I checked the error in details and it would be because the Instance Profile Name and the Role ARN name don't match exactly. Please see points 3 and 4 here in the docs: https://docs.databricks.com/sql/admin/serverle...
- 0 kudos
- 3332 Views
- 1 replies
- 0 kudos
CDC DLT
Hi all,I would appreciate some clarity regarding the DLT and CDC. So my first question would be, when it comes to the "source" table in the synta, is that CDC table or? Further, if we want to use only databricks, would mounting foreign catalog be a g...
- 3332 Views
- 1 replies
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 581 | |
| 349 | |
| 287 |
