- 3018 Views
- 2 replies
- 2 kudos
Resolved! Where is Databricks Tunnel (and is Databricks connect cool again?)
Two related questions:1: There has been several mentions in this forum about "Databricks Tunnel", which should allow us to connect from our local IDE to a remote databricks cluster and develop stuff locally. The roumors said early 2022, is there some...
- 3018 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi there @Erik Parmann​ Does @Youssef Mrini​ response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks
- 2 kudos
- 2369 Views
- 3 replies
- 1 kudos
Any plans to provide Databricks SQL / Alerts API
Hi, Databricks! You are my favorite Big Data tool, but I've recently faced an issue I didn't expect to have. For our agriculture customers, we're trying to use Databricks SQL Platform to keep our data accurate all day. We use Alerts to validate our d...
- 2369 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Dmytro Imshenetskyi​ Does @Hubert Dudek​ response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!
- 1 kudos
- 1655 Views
- 1 replies
- 0 kudos
Cannot sign in at databricks partner-academy portal
Hi thereI have used my company email to register an account for customer-academy.databricks.com a while back.Now what I need to do is create an account with partner-academy.databricks.com using my company email too.However when I register at partner-...
- 1655 Views
- 1 replies
- 0 kudos
- 4896 Views
- 2 replies
- 1 kudos
Resolved! Error deleting a table
I'm trying to delete a table that was created from a csv and due to the file deletion, I can't execute the deletion, with the following error: I'm new to Databricks and I don't know how to fix this. Some help?
- 4896 Views
- 2 replies
- 1 kudos
- 1 kudos
To delete the table, it's looking for underlying delta log file and because the file doesn't exist, it's throwing you that error.Just drop the table.drop table <table_name>
- 1 kudos
- 3308 Views
- 4 replies
- 7 kudos
Resolved! Azure data bricks delta tables .Issue
Hello Team,I have written Spark SQL Query in data bricks :DROP TABLE IF EXISTS Salesforce.Location;CREATE EXTERNAL TABLE Salesforce.Location (Id STRING,OwnerId STRING,IsDeleted bigint,Name STRING,CurrencyIsoCode STRING,CreatedDate bigint,CreatedById ...
- 3308 Views
- 4 replies
- 7 kudos
- 7 kudos
You need to provide one of the following value for 'data_source':TEXTAVROCSVJSONPARQUETORCDELTAeg: USING PARQUETIf you skip USING clause, then the default data source is DELTAhttps://docs.databricks.com/sql/language-manual/sql-ref-syntax-ddl-create-t...
- 7 kudos
- 9219 Views
- 6 replies
- 4 kudos
Resolved! Lesson 6.1 of Data Engineering. Error when reading stream - java.lang.UnsupportedOperationException: com.databricks.backend.daemon.data.client.DBFSV1.resolvePathOnPhysicalStorage(path: Path)
Below function executes fine: def autoload_to_table(data_source, source_format, table_name, checkpoint_directory): query = (spark.readStream .format("cloudFiles") .option("cloudFiles.format", source_format) .option("cloudFile...
- 9219 Views
- 6 replies
- 4 kudos
- 3697 Views
- 2 replies
- 2 kudos
Resolved! Delta Live Tables not inferring table schema properly.
I have a delta live tables pipeline that is loading and transforming data. Currently I am having a problem that the schema inferred by DLT does not match the actual schema of the table. The table is generated via a groupby.pivot operation as follows:...
- 3697 Views
- 2 replies
- 2 kudos
- 2 kudos
I was able to get around this by specifying the table schema in the table decorator.
- 2 kudos
- 1749 Views
- 1 replies
- 0 kudos
Optimal approach when using external script/executable for processing data
I need to process a number of files where I manipulate file text utilising an external executable that operates on stdin/stdout. I am quite new to spark. What I am attempting is to use rdd.pipe as in the followingexe_path = " /usr/local/bin/external...
- 1749 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Michael Lennon​ Can you please elaborate use case on what the external app is doing exe_path
- 0 kudos
- 3727 Views
- 2 replies
- 4 kudos
How to set up partitions on the streaming Delta Table?
Let's assume that we have 3 streaming Delta Tables:BronzeSilverGoldMy aim is to add partitioning to Silver table (for example by Date). So, as a result Gold table with throw an error that source table has been updated and I would need to set 'ignoreC...
- 3727 Views
- 2 replies
- 4 kudos
- 4 kudos
is the change data feed functionality (of your silver table) an option, combined with merge in your gold table?https://docs.microsoft.com/en-us/azure/databricks/delta/delta-change-data-feed
- 4 kudos
- 11941 Views
- 1 replies
- 1 kudos
Resolved! Databricks - python error when importing wheel distribution package
In previous days all notebooks containing : 'import anomalydetection' worked just fine. There was no change in any configuration of the cluster, notebook or our imported library.However recently notebooks just crashed with below errorSame happen also...
- 11941 Views
- 1 replies
- 1 kudos
- 1 kudos
Solution: This is due to the latest version of protobuf library, please try to downgrade the library which should solve the issuepip install protobuf==3.20.*protobuf library versions which works: 3.20.1 if it does not work then try 3.18.1
- 1 kudos
- 2091 Views
- 0 replies
- 0 kudos
How to use Terraform to add Git provider credentials to a workspace in order to use service principal for CI/CD
Hi,I'm very new to Terraform. Currently, I'm trying to automate the service principal setup process using Terraform.Following this example, I successfully created a service principal and an access token. However, when I tried adding databricks_git_cr...
- 2091 Views
- 0 replies
- 0 kudos
- 5206 Views
- 2 replies
- 0 kudos
spark.read.parquet() - how to check for file lock before reading? (azure)
I have some python code which takes parquet files from an adlsv2 location and merges it into delta tables (run as a workflow job on a schedule)I have a try catch wrapper around this so that any files that fail get moved into a failed folder using dbu...
- 5206 Views
- 2 replies
- 0 kudos
- 0 kudos
That's the problem - it's not being locked (or fs.mv() isn't checking/honoring the lock). The upload process/tool is a 3rd-prty external toolI can see via the upload tool that the file upload is 'in progress'I can also see the 0 byte destination file...
- 0 kudos
- 10314 Views
- 8 replies
- 3 kudos
Resolved! Are MERGE INTO inserts supported when the delta table has an identity column ?
I can't seem to make it work as I keep getting:DeltaInvariantViolationException: NOT NULL constraint violated for column: dl_id.
- 10314 Views
- 8 replies
- 3 kudos
- 3 kudos
if you are using 'delta.columnMapping.mode' = 'name' on your table i could not get it to work, without that line .. for the not matched .. WHEN NOT MATCHED THEN INSERT (columnname,columnName2) values(columnname,columnName2)WHEN MATCHED Then UPDAT...
- 3 kudos
- 1779 Views
- 0 replies
- 2 kudos
Moving Notebook Cell causes browser to run out of memory
Platform:AWS Databricksenabled "Turn on the new, updated code editor" in Notebook SettingsMacOS 12.5.1Firefox 104.0.2When I attempt to drag a notebook cell to move it, the tab crashes and causes my computer to run out of memory. I profiled the tab to...
- 1779 Views
- 0 replies
- 2 kudos
- 2131 Views
- 1 replies
- 3 kudos
Register for Databricks Office HoursSeptember 14: 8:00 - 9:00 AM PT | 3:00pm - 4:00pm GMTSeptember 28: 11:00 AM - 12:00 PM PT | 6:00 - 7:00 PM GMTÂ Dat...
Register for Databricks Office HoursSeptember 14: 8:00 - 9:00 AM PT | 3:00pm - 4:00pm GMTSeptember 28: 11:00 AM - 12:00 PM PT | 6:00 - 7:00 PM GMTDatabricks Office Hours connects you directly with experts to answer your Databricks questions.Join us t...
- 2131 Views
- 1 replies
- 3 kudos
- 3 kudos
Check out some of the questions from fellow users during our last Office Hours. All these questions were answered live by a Databricks expert!Q: What's the best way of using a UDF in a class?A: You need to define your class and then register the func...
- 3 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 553 | |
| 349 | |
| 287 |
