- 12944 Views
- 10 replies
- 5 kudos
Resolved! An unidentified special character is added in outbound file when transformed in databricks. Please help with suggestion?
Data from external source is copied to ADLS, which further gets picked up by databricks, then this massaged data is put in the outbound file . A special character ? (question mark in black diamond) is seen in some fields in outbound file which may br...
- 12944 Views
- 10 replies
- 5 kudos
- 5 kudos
Are you sure it is Databricks which puts the special character in place?It could also have happened during the copy of the external system to ADLS.If you use Azure Data Factory f.e. you have to define the encoding (UTF-8 or UTF-16, ...)
- 5 kudos
- 18874 Views
- 8 replies
- 9 kudos
Resolved! Is there a way to create a non-temporary Spark View with PySpark?
Hi,When creating a Spark view using SparkSQL ("CREATE VIEW AS SELCT ...") per default, this view is non-temporary - the view definition will survive the Spark session as well as the Spark cluster.In PySpark I can use DataFrame.createOrReplaceTempView...
- 18874 Views
- 8 replies
- 9 kudos
- 9 kudos
why not to create manage table?dataframe.write.mode(SaveMode.Overwrite).saveAsTable("<example-table>") # later when we need data resultDf = spark.read.table("<example-table>")
- 9 kudos
- 8712 Views
- 9 replies
- 4 kudos
Resolved! Does Databricks persist the alert results history?
Hi, this seems like a really basic feature to have, an alert is generated and spits out the email, but the URL doesn't take you to the events list that happened at that time but just the query in the editor (via the alert config screen). We're really...
- 8712 Views
- 9 replies
- 4 kudos
- 4 kudos
@Nick Hughes​ I was looking at our ideas portal and could see an API feature was requested for the Alerts (DB-I-4289). To create an API for the alerts and pull data, we might need to keep the alert history persistent. So this feature should suffice y...
- 4 kudos
- 1363 Views
- 0 replies
- 4 kudos
spark.apache.org
Per API docs on StreamingQuery.stop(), https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/streaming/StreamingQuery.html It says, this stops the execution of this query if it is running and waits until the termination of the query...
- 1363 Views
- 0 replies
- 4 kudos
- 10707 Views
- 5 replies
- 6 kudos
Resolved! Autoloader failed
I used autoloader with TriggerOnce = true and ran it for weeks with schedule. Today it broke:The metadata file in the streaming source checkpoint directory is missing. This metadatafile contains important default options for the stream, so the stream...
- 10707 Views
- 5 replies
- 6 kudos
- 6 kudos
Hi dimoobraznii (Customer)This error comes in streaming when someone makes changes to the streaming checkpoint directory manually or points some streaming type to the checkpoint of some other streaming type. Please check if any changes were made to t...
- 6 kudos
- 4738 Views
- 3 replies
- 7 kudos
Resolved! How does 73% of the data go unused for analytics or decision-making?
Is Lakehouse the answer? Here's a good resource that was just published: https://dbricks.co/3q3471X
- 4738 Views
- 3 replies
- 7 kudos
- 7 kudos
@Alexis Lopez​ - If @Dan Zafar​ 's or @Harikrishnan Kunhumveettil​'s answers solved the issue, would you be happy to mark one of their answers as best so other members can find the solution more easily?
- 7 kudos
- 2437 Views
- 1 replies
- 1 kudos
Auto Loader Schema-Inference and Evolution for parquet files
Dear all,will (and when) will Auto Loader also support Schema-Inference and Evolution for parquet files, at this point it is only for JSON and CSV supported if i am not mistaken?Thanks and regards,Gapy
- 2437 Views
- 1 replies
- 1 kudos
- 1 kudos
@Gasper Zerak​ , This will be available in near future (DBR 10.3 or later). Unfortunately, we don't have an SLA at this moment.
- 1 kudos
- 9241 Views
- 10 replies
- 9 kudos
Resolved! Lineage between model and source code breaks on movement of source notebook. How to rectify it?
If there is a registered model and it is linked with a notebook, then the lineage breaks if you move the notebook to a different path or even pull/upload a new version of the notebook.This is not good because when someone doing its development/testin...
- 9241 Views
- 10 replies
- 9 kudos
- 9 kudos
I also cannot reproduce this, with these exact steps (I think). After moving the notebook and moving it back, the link to it (and link to the revision) still works as expected. You are using MLflow built in to Databricks right?
- 9 kudos
- 9002 Views
- 1 replies
- 3 kudos
Resolved! %run file not found
Hi,I was using the following command to import variables and functions from an other notebook :%run ./utilsFor some reason it is not working any more and gives me this message :Exception: File `'./utils.py'` not found.utils.py is still at the same pl...
- 9002 Views
- 1 replies
- 3 kudos
- 3 kudos
Finally I just solved my issue.Actually, in the same cell I wrote a comment starting with # and it was not working because of that...
- 3 kudos
- 14327 Views
- 4 replies
- 3 kudos
Is it possible to access a variable in markdown cells?
I saw a similar question in a discussion 5 years ago and back then this option was not available. https://community.databricks.com/s/question/0D53f00001HKHhNCAX/markup-in-databricks-notebookI wonder if this feature has been added. It is possible to d...
- 14327 Views
- 4 replies
- 3 kudos
- 3 kudos
When I click on the feedback link under workspace/help, it opens a page with page not found.
- 3 kudos
- 7984 Views
- 4 replies
- 3 kudos
Resolved! Location of customer_t1 dataset
Can anyone tell me how I can access the customer_t1 dataset that is referenced in the book "Delta Lake - The Definitive Guide "? I am trying to follow along with one of the examples.
- 7984 Views
- 4 replies
- 3 kudos
- 3 kudos
Some files are visualized here https://github.com/vinijaiswal/delta_time_travel/blob/main/Delta%20Time%20Travel.ipynb but it is quite strange that there is no source in repository. I think only one way is to write to Vini Jaiswal on github.
- 3 kudos
- 1726 Views
- 1 replies
- 2 kudos
docs.databricks.com
Are EBS volumes used by Databricks Clusters are encrypted especially the root volumes
- 1726 Views
- 1 replies
- 2 kudos
- 2 kudos
Yes these EBS volumes are encrypted. Earlier root volume encryptions were not supported but recently this encryption is also enabled (since Apr, 2021)please find more details on the below docs pagehttps://docs.databricks.com/clusters/configure.html#e...
- 2 kudos
- 8483 Views
- 6 replies
- 5 kudos
Resolved! /dbfs is empty
Why does /dbfs seem to be empty in my Databricks cluster ?If I run %sh ls /dbfsI get no output.I am looking for the databricks-datasets subdirectory ? I can't find it under /dbfs
- 8483 Views
- 6 replies
- 5 kudos
- 5007 Views
- 3 replies
- 2 kudos
Resolved! log error to cosmos db
Objective:- Retrieve objects from an S3 bucket using a 'get' api call, write the retrieved object to azure datalake and in case of errors like 404s (object not found) write the error message to cosmos DB"my_dataframe" consists of the a column (s3Obje...
- 5007 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @Sandesh Puligundla​ issue is that you are using spark context inside foreachpartition. You can create a dataframe only on the spark driver. Few stack overflow references https://stackoverflow.com/questions/46964250/nullpointerexception-creatin...
- 2 kudos
- 4299 Views
- 2 replies
- 4 kudos
Resolved! Identifying the right tools for the job
Hi all, thank you for taking the time to attend to my post. A background to preface, my team and I have been prototyping an ML model that we would like to push into the production and deployment phase. We have been prototyping on Jupyter Notebooks bu...
- 4299 Views
- 2 replies
- 4 kudos
- 4 kudos
For production model serving, why not just use MLflow Model Serving? You just code it up/import it with the notebooks, then Log it using MLflow, then Register it with the MLflow Registry, then there is a nice UI to serve it using Model Serving. It wi...
- 4 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
