- 3471 Views
- 2 replies
- 1 kudos
Can't import st_constructors module after installing Apache Sedona
Hi there,I've recently installed Apache Sedona on my cluster, according to the detailed instructions here. My Databricks runtime version is 9.1 LTS (includes Apache Spark 3.1.2, Scala 2.12).The installation included the apache-sedona library from PyP...
- 3471 Views
- 2 replies
- 1 kudos
- 1 kudos
Thank you @Debayan Mukherjee​ for the prompt reply. I've followed the instructions carefully, but now every time I try to run a cell in my notebook I get a "Cancelled" message. It clearly didn't work. Any advice?Your help is much appreciated.
- 1 kudos
- 13464 Views
- 5 replies
- 1 kudos
- 13464 Views
- 5 replies
- 1 kudos
- 1 kudos
Any string between double curly braces {{ }} is treated as a query parameter. widget can be an option here.
- 1 kudos
- 13887 Views
- 2 replies
- 6 kudos
Schema Evolution Issue in Streaming
When there is a schema change while reading and writing to a stream, will the schema changes be automatically handled by sparkor do we need to include the option(mergeschema=True)?Eg:df.writeStream .option("mergeSchema", "true") .format("delta") .out...
- 13887 Views
- 2 replies
- 6 kudos
- 6 kudos
mergeSchema doesn't support all operations. In some cases .option("overwriteSchema", "true") is needed. MergeSchema doesn't support:Dropping a columnChanging an existing column's data type (in place)Renaming column names that differ only by case (e.g...
- 6 kudos
- 42410 Views
- 3 replies
- 5 kudos
Resolved! Parse Syntax error ,can anyone guide me what is going wrong here
Select case WHEN {{ Month }} = 0 then add_months(current_date(),-13 ) elseWHEN {{ Month }}> month(add_months(current_date(),-1)) then add_months(to_date(concat(year(current_date())-1,'-',{{Month}},'-',1)),-13) else add_months(to_date(conc...
- 42410 Views
- 3 replies
- 5 kudos
- 5 kudos
Hi @Sajid Thavalengal Rahiman​ , Have you followed the recommendation given above? Also, could you please paste the whole error with the code?
- 5 kudos
- 1725 Views
- 0 replies
- 1 kudos
Voucher sending date
Completed the attached course today Nov 3rd. In order to avoid writing again and requesting a voucher, when is the voucher for this one expected to be sent?Thanks
- 1725 Views
- 0 replies
- 1 kudos
- 5124 Views
- 1 replies
- 0 kudos
Resolved! How do you run a query as the owner but use a parameter as a viewer
I have a query that is hitting a table I have access too. Granting access to everyone is not an option. I am using this query in a SQL Dashboard. One of the where clause conditions uses a parameter populated by another query. I want this parameter qu...
- 5124 Views
- 1 replies
- 0 kudos
- 0 kudos
It is not possible to do what I want. Somewhat seems like a security flaw but what ever
- 0 kudos
- 5373 Views
- 3 replies
- 2 kudos
Resolved! Orchestrate run of a folder
I'm needing to run the contents of a folder, which can change over time. Is there a way to set up a notebook that can orchestrate running all notebooks in a folder? My though was if I could retrieve a list of the notebooks I could create a loop to ru...
- 5373 Views
- 3 replies
- 2 kudos
- 2 kudos
List all notebooks by making API call and then run them by using dbutils.notebook.run:import requests ctx = dbutils.notebook.entry_point.getDbutils().notebook().getContext() host_name = ctx.tags().get("browserHostName").get() host_token = ctx.apiToke...
- 2 kudos
- 8256 Views
- 5 replies
- 5 kudos
Resolved! How do I clone a repo in Community Edition?
The e-learning videos on DBacademy say we should click on "Repos" and "Add Repo"I cannot find this in my Community Edition UII am a little frustrated that there are so many different versions of the UI and many videos show UI options that we cannot ...
- 8256 Views
- 5 replies
- 5 kudos
- 5 kudos
Hello, just import the .dbc file direct into your user workspace, as explained by Databricks here:https://www.databricks.training/step-by-step/importing-courseware-from-github/The simplest way
- 5 kudos
- 75878 Views
- 3 replies
- 9 kudos
Best practice for logging in Databricks notebooks?
What is the best practice for logging in Databricks notebooks? I have a bunch of notebooks that run in parallel through a workflow. I would like to keep track of everything that happens such as errors coming from a stream. I would like these logs to ...
- 75878 Views
- 3 replies
- 9 kudos
- 9 kudos
@Gimwell Young​ AS @Debayan Mukherjee​ mentioned if you configure verbose logging in workspace level, logs will be moved to your storage bucket that you have provided during configuration. from there you can pull logs into any of your licensed log mo...
- 9 kudos
- 6081 Views
- 7 replies
- 0 kudos
Issue on Cluster creating new workspace: I Cannot able to create a new workspace in Databricks using Quickstart. When I am creating the workspace I ge...
Issue on Cluster creating new workspace: I Cannot able to create a new workspace in Databricks using Quickstart. When I am creating the workspace I get the Rollback failed error from AWS eventhoughI have given all the required informations. Kindly he...
- 6081 Views
- 7 replies
- 0 kudos
- 0 kudos
hi @Gopichandran N​ could you please add more information on the issue that you are facing. could you please add the screenshot of the error?
- 0 kudos
- 3654 Views
- 2 replies
- 17 kudos
Autoloader: how to avoid overlap in files
I'm thinking of using autoloader to process files being put on our data lake.Let's say f.e. every 15 minutes, a parquet files is written. These files however contain overlapping data.Now, every 2 hours I want to process the new data (autoloader) and...
- 3654 Views
- 2 replies
- 17 kudos
- 17 kudos
What about forEachBatch and then MERGE?Alternatively, run another process that will clean updates using the window function, as you said.
- 17 kudos
- 4906 Views
- 1 replies
- 7 kudos
Move folder from dbfs location to user workspace directory in azure databricks
I need to move group of files(python or scala file from)or folder from dbfs location to user workspace directory in azure databricks to do testing on file.Its verify difficult to upload each file one by one into the user workspace directory, so is it...
- 4906 Views
- 1 replies
- 7 kudos
- 7 kudos
dbutils.fs.mv or dbutils.fs.cp can help you.
- 7 kudos
- 4620 Views
- 3 replies
- 13 kudos
Resolved! SCD type 2
Hey guys. I don't know if I'm tired, I ask for your help, but I don't understand where is the difference in the number of fields.Thanks! I'm replicating SCD type 2 based on this documentation:https://docs.delta.io/latest/delta-update.html#slowly-chan...
- 4620 Views
- 3 replies
- 13 kudos
- 8166 Views
- 0 replies
- 2 kudos
How to check, who did changes in databrics notebook and what are changes done in databrics notebook.
How to check, who did changes in databrics notebook and what are changes done in databrics notebook.
- 8166 Views
- 0 replies
- 2 kudos
- 3724 Views
- 2 replies
- 3 kudos
Resolved! to configure Autoloader in File notification mode to access the Premium BlobStorage
First, I tried to configure Autoloader in File notification mode to access the Premium BlobStorage 'databrickspoc1' (PREMIUM , ADLS Gen2). I get this Error: I get this errorcom.microsoft.azure.storage.StorageException: I checked my storage account->N...
- 3724 Views
- 2 replies
- 3 kudos
- 3 kudos
When you created a premium account, have you chosen "Premium account type" as "File shares"? It should be "Block blobs".
- 3 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 561 | |
| 349 | |
| 287 |
