- 2785 Views
- 2 replies
- 3 kudos
Databricks streaming dataframe into Snowflake
Any suggestions on how to stream data from databricks into snowflake?. Is snowpipe is the only option?. Snowpipe is not faster since it runs copy into in a small batch intervals and not in few seconds. If no option other than snowpipe, how to call it...
- 2785 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @Anonymous Hope everything is going great. Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we...
- 3 kudos
- 8494 Views
- 10 replies
- 7 kudos
DLT optimize and vacuum
We were finally able to get DLT pipelines to run the optimize and vacuum automatically. We verified this via the the table history. However I am able to still query versions older than 7 days. Has anyone been experiencing this and how were you a...
- 8494 Views
- 10 replies
- 7 kudos
- 7 kudos
Hi @Gil Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you. Thanks!
- 7 kudos
- 3151 Views
- 2 replies
- 5 kudos
Dear @Werner Stinckens and @Tyler Retzlaff We would like to express our gratitude for your participation and dedication in the Databricks Commun...
Dear @Werner Stinckens and @Tyler Retzlaff We would like to express our gratitude for your participation and dedication in the Databricks Community last week. Your interactions with customers have been valuable and we truly appreciate the time...
- 3151 Views
- 2 replies
- 5 kudos
- 9910 Views
- 2 replies
- 4 kudos
Resolved! How does merge schema work
Let's say I create a table like CREATE TABLE IF NOT EXISTS new_db.data_table ( key STRING, value STRING, last_updated_time TIMESTAMP ) USING DELTA LOCATION 's3://......';Now when I insert into this table I insert data which has say 20 columns a...
- 9910 Views
- 2 replies
- 4 kudos
- 4 kudos
I tried running "REFRESH TABLE tablename;" but I still do not see the added columns in the data explorer columns, while I do see the added columns in the sample data
- 4 kudos
- 4498 Views
- 4 replies
- 0 kudos
_sqldf value in case of query failure in %sql cell
I am trying to write a code for Error Handling in Databricks notebook in case of a SQL magic cell failure. I have a %sql cell followed by some python code in next cells. I want to abort the notebook if the query in %sql cell fails. To do so I am look...
- 4498 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @pjain We haven't heard from you since the last response from @daniel_sahal , and I was checking back to see if her suggestions helped you. Or else, If you have any solution, please share it with the community, as it can be helpful to others. A...
- 0 kudos
- 23574 Views
- 15 replies
- 5 kudos
Resolved! Lost memory when using dbutils
Why does copying a 9GB file from a container to the /dbfs lose me 50GB of memory? (Which doesn't come back until I restarted the cluster)
- 23574 Views
- 15 replies
- 5 kudos
- 5 kudos
Hi James,Did you get to the bottom of this? We are experiencing the same issue, and all the suggested solutions don't seem to work.Thanks,Adrian
- 5 kudos
- 4365 Views
- 3 replies
- 3 kudos
Resolved! Error on Azure-Databricks write RDD to storage account with wsabs://
Hi, I'm trying to write data from RDD to the storage account:Adding storage account key:spark.conf.set("fs.azure.account.key.y.blob.core.windows.net", "myStorageAccountKey")Read and write to the same storage:val path = "wasbs://x@y.blob.core.windows....
- 4365 Views
- 3 replies
- 3 kudos
- 3 kudos
Hello @Vadim1 and @User16764241763. I'm wondering if you find a way to avoid adding the hardcoded key in the advanced options spark config section in the cluster configuration. Is there a similar command to spark.conf.set("spark.hadoop.fs.azure.accou...
- 3 kudos
- 10647 Views
- 8 replies
- 11 kudos
As the Databricks account manager; I cannot login: "Your user already belongs to a Databricks account"
On GCP I subscribed to Databricks in one project within the organization.Then I canceled this subscription and subscribed to Databricks in another project.When I try to login onto the newly subscribed databricks with google SSO:> There was an error s...
- 10647 Views
- 8 replies
- 11 kudos
- 11 kudos
I can see the issue might be related to organizations or billing accounts. The new Databricks project I tried creating was on a different organization/billing-account than the test Databricks subscription I created a month back.I went back to the ori...
- 11 kudos
- 13916 Views
- 4 replies
- 6 kudos
Resolved! Location not empty but not a Delta table
I need help or insight regarding the following errors. My instructors (Brooke Wenig with Conor Murphy) ran this code successfully on our course video, but I cannot replicate what she did. Here is the code and below it is the outcome from my Cours...
- 13916 Views
- 4 replies
- 6 kudos
- 6 kudos
DELETE the original Parquet table as a separate statementHow can this be done? simple query "DROP TABLE .... " still failing with "cannot be found"
- 6 kudos
- 781 Views
- 0 replies
- 0 kudos
Permission on Unity catalog
I am facing an issue when assign permission on view created on unity catalog. The problem is I had create a user defined function (UDFs) in order to encrypt sensitive column, I create a view which call the functions and source table within the catalo...
- 781 Views
- 0 replies
- 0 kudos
- 12415 Views
- 7 replies
- 7 kudos
Resolved! Accessing workspace files within cluster init script
Greetings all!I am currently facing an issue while accessing workspace files from the init script.As it was explained in the documentation, it is possible to place init script inside workspace files (link). This works perfectly fine and init script i...
- 12415 Views
- 7 replies
- 7 kudos
- 7 kudos
@Gleb Smolnik You might also want to try cloning a github repo in your init script and then storing dependencies like requirements.txt files and other init scripts there. By doing this you can pull a whole slew of init scripts to be utilized in your...
- 7 kudos
- 5554 Views
- 4 replies
- 8 kudos
Resolved! spark managed tables
Hi, I recently started learning about spark. I was studying about spark managed tables. so as per docs " spark manages the both the data and metadata". Assume that i have a csv file in s3 and I read it into data frame like below.df = spark.read .for...
- 5554 Views
- 4 replies
- 8 kudos
- 8 kudos
Yes, @Raviiit DBFS (Databricks File System) is a distributed file system used by Databricks clusters. DBFS is an abstraction layer over cloud storage (e.g. S3 or Azure Blob Store), allowing external storage buckets to be mounted as paths in the DBFS ...
- 8 kudos
- 7857 Views
- 5 replies
- 0 kudos
File copy in adls
i am using dbutils.fs.copy(abfss://container/provsn/filen[ame.txt,abfss://container/data/sasam.txt)while.trying this copy method to copy the files it is showing urisyntax exception near the square bracket how can i read and copy it
- 7857 Views
- 5 replies
- 0 kudos
- 0 kudos
From looking at stack trace, it looks like URIException. Easiest solution would be renaming the file so that there are no square brackets in the filename. If this is not an option, it might help to URLEncode the path - https://stackoverflow.com/que...
- 0 kudos
- 5806 Views
- 3 replies
- 2 kudos
Passing values between notebook tasks in Workflow Jobs
I have created a Databricks workflow job with notebooks as individual tasks sequentially linked. I assign a value to a variable in one notebook task (ex: batchid = int(time.time()). Now, I want to pass this batchid variable to next notebook task.What...
- 5806 Views
- 3 replies
- 2 kudos
- 2 kudos
@brickster You would use dbutils.jobs.taskValues.set() and dbutils.jobs.taskValues.get().See docs for more details: https://docs.databricks.com/workflows/jobs/share-task-context.html
- 2 kudos
- 9089 Views
- 6 replies
- 1 kudos
Resolved! On-Premise SQL Server Ingestion to Databricks Bronze Layer
Hello everyone!So I want to ingest tables with schemas from the on-premise SQL server to Databricks Bronze layer with Delta Live Table and I want to do it using Azure Data Factory and I want the load to be a Snapshot batch load, not an incremental lo...
- 9089 Views
- 6 replies
- 1 kudos
- 1 kudos
Hi @Parsa Bahraminejad Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best an...
- 1 kudos
| User | Count |
|---|---|
| 1616 | |
| 790 | |
| 466 | |
| 349 | |
| 287 |
