- 14415 Views
- 6 replies
- 3 kudos
Resolved! Is there a way to run a single or selected lines in a notebook?
Assume we have a given cellprint('A') print('B') print('C')I want to run only the below line.print('B')Obviously, I can seperate the cell into three and run the one I want, but this is timely. This is a feature I use so often (e.g. in pycharm) and wo...
- 14415 Views
- 6 replies
- 3 kudos
- 3 kudos
@Volkan_Gumuskay This is also available as an option in the notebook run options.
- 3 kudos
- 6777 Views
- 2 replies
- 3 kudos
Row_Num function in spark-sql
I have a doubt row_num with order by in spark-sql gives different result(non-deterministic output) every time i execute it?It's due to parallelism in spark ?Any approach how to takle it?I order by with a date column and a integer column and take...
- 6777 Views
- 2 replies
- 3 kudos
- 3 kudos
@Hemant If the order by clause provided yields a unique result, then we would get deterministic output. For ex:If we create a rowID for this dataset, with CustomerID used in OrderBy clause, then depending upon the runtime, we may get non-deterministi...
- 3 kudos
- 15877 Views
- 3 replies
- 0 kudos
Resolved! Merge Schema Error Message despite setting option to true
Has anyone come across this error before:```A schema mismatch detected when writing to the Delta table (Table ID: d4b9c839-af0b-4b62-aab5-1072d3a0fa9d). To enable schema migration using DataFrameWriter or DataStreamWriter, please set: '.option("merge...
- 15877 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @alexiswl Share the wisdom! By marking the best answers, you help others in our community find valuable information quickly and efficiently. Thanks!
- 0 kudos
- 3566 Views
- 2 replies
- 0 kudos
- 3566 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Yogybricks Hope you are well. Just wanted to see if you were able to find an answer to your question and would you like to mark an answer as best? It would be really helpful for the other members too. Cheers!
- 0 kudos
- 5621 Views
- 2 replies
- 0 kudos
Resolved! Trigger file_arrival of job on Delta Lake table change
Is there a way to avoid having to create an external data location Simply to trigger a job when new data comes to a specific Delta Lake table?
- 5621 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @zsucic1 Hope you are well. Just wanted to see if you were able to find an answer to your question and would you like to mark an answer as best? It would be really helpful for the other members too. Cheers!
- 0 kudos
- 6828 Views
- 3 replies
- 0 kudos
Not able to execute below SQL query in databricks notebook because of Pare error
Hi Team,I am unable to run the below command and it is giving me a parse error. Can any one point out the issue with the code:
- 6828 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi,From the error , it looks like there is no space between the brackets and the "in" keyword after the where clause. Can you please try again see if you facing the same error.
- 0 kudos
- 3913 Views
- 2 replies
- 1 kudos
Consume gold data layer from web application
Hello!We are developing a web application in .NET, we need to consume data in gold layer, (as if we had a relational database), how can we do it? export data to sql server from gold layer?
- 3913 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @apiury Thank you for posting your question in our community! We are happy to assist you. To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your ...
- 1 kudos
- 33644 Views
- 12 replies
- 0 kudos
How can I split a Spark Dataframe into n equal Dataframes (by rows)? I tried to add a Row ID column to acheive this but was unsuccessful.
I have a dataframe that has 5M rows. I need to split it up into 5 dataframes of ~1M rows each. This would be easy if I could create a column that contains Row ID. Is that possible?
- 33644 Views
- 12 replies
- 0 kudos
- 0 kudos
Hi @NithinTiruveedh Thank you for posting your question in our community! We are happy to assist you. To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answ...
- 0 kudos
- 1589 Views
- 0 replies
- 0 kudos
Higher Order Function: AGGREGATE not working in the example notebook mentioned in Documentation
Hi All,I am running a sample notebook from Databricks Documentation section on Higher Order Function on my community edition. I am running this notebook on DBR 12.2 LTS.Databricks Documentation URL : https://docs.databricks.com/optimizations/higher-o...
- 1589 Views
- 0 replies
- 0 kudos
- 3637 Views
- 2 replies
- 3 kudos
Databricks streaming dataframe into Snowflake
Any suggestions on how to stream data from databricks into snowflake?. Is snowpipe is the only option?. Snowpipe is not faster since it runs copy into in a small batch intervals and not in few seconds. If no option other than snowpipe, how to call it...
- 3637 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @Anonymous Hope everything is going great. Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we...
- 3 kudos
- 10607 Views
- 10 replies
- 7 kudos
DLT optimize and vacuum
We were finally able to get DLT pipelines to run the optimize and vacuum automatically. We verified this via the the table history. However I am able to still query versions older than 7 days. Has anyone been experiencing this and how were you a...
- 10607 Views
- 10 replies
- 7 kudos
- 7 kudos
Hi @Gil Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you. Thanks!
- 7 kudos
- 3824 Views
- 2 replies
- 5 kudos
Dear @Werner Stinckens and @Tyler Retzlaff We would like to express our gratitude for your participation and dedication in the Databricks Commun...
Dear @Werner Stinckens and @Tyler Retzlaff We would like to express our gratitude for your participation and dedication in the Databricks Community last week. Your interactions with customers have been valuable and we truly appreciate the time...
- 3824 Views
- 2 replies
- 5 kudos
- 11287 Views
- 2 replies
- 4 kudos
Resolved! How does merge schema work
Let's say I create a table like CREATE TABLE IF NOT EXISTS new_db.data_table ( key STRING, value STRING, last_updated_time TIMESTAMP ) USING DELTA LOCATION 's3://......';Now when I insert into this table I insert data which has say 20 columns a...
- 11287 Views
- 2 replies
- 4 kudos
- 4 kudos
I tried running "REFRESH TABLE tablename;" but I still do not see the added columns in the data explorer columns, while I do see the added columns in the sample data
- 4 kudos
- 5516 Views
- 4 replies
- 0 kudos
_sqldf value in case of query failure in %sql cell
I am trying to write a code for Error Handling in Databricks notebook in case of a SQL magic cell failure. I have a %sql cell followed by some python code in next cells. I want to abort the notebook if the query in %sql cell fails. To do so I am look...
- 5516 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @pjain We haven't heard from you since the last response from @daniel_sahal , and I was checking back to see if her suggestions helped you. Or else, If you have any solution, please share it with the community, as it can be helpful to others. A...
- 0 kudos
- 30647 Views
- 15 replies
- 5 kudos
Resolved! Lost memory when using dbutils
Why does copying a 9GB file from a container to the /dbfs lose me 50GB of memory? (Which doesn't come back until I restarted the cluster)
- 30647 Views
- 15 replies
- 5 kudos
- 5 kudos
Hi James,Did you get to the bottom of this? We are experiencing the same issue, and all the suggested solutions don't seem to work.Thanks,Adrian
- 5 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 581 | |
| 349 | |
| 287 |
