- 18889 Views
- 1 replies
- 0 kudos
I would like to know the difference between .mode("append") and .mode("overwrite") when writing my Delta table
- 18889 Views
- 1 replies
- 0 kudos
Latest Reply
Mode "append" atomically adds new data to an existing Delta table and "overwrite" atomically replaces all of the data in a table.
- 1680 Views
- 1 replies
- 0 kudos
I would like to know where can I find the current schema information from my Delta table.
- 1680 Views
- 1 replies
- 0 kudos
Latest Reply
The table name, path, database info are stored in Hive metastore, the actual schema is stored in the "_delta_log" directory that should be in the root path location where you Delta table is stored.
- 4730 Views
- 1 replies
- 0 kudos
is there a way to read a specific part off a delta table? When I try to read the parquet file as parquet I get an error in the notebook that I’m using the incorrect format as it’s part of a delta table. I just want to read a single Parquet file, not ...
- 4730 Views
- 1 replies
- 0 kudos
Latest Reply
Disable Delta format to read as Parquet you need to set to false the following Spark settings:>> SET spark.databricks.delta.formatCheck.enabled=false OR>> spark.conf.set("spark.databricks.delta.formatCheck.enabled", "false")its not recommended to re...
- 1430 Views
- 1 replies
- 0 kudos
I would like to know if it recommended to run Analyze table on Delta tables or not. If not, why?
- 1430 Views
- 1 replies
- 0 kudos
Latest Reply
You can run ANALYZE TABLE on Delta tables only on Databricks Runtime 8.3 and above. For more details please refer to the docs: https://docs.databricks.com/spark/latest/spark-sql/language-manual/sql-ref-syntax-aux-analyze-table.html
- 1068 Views
- 1 replies
- 1 kudos
We are trying to download our repository which is hosted on GitHub Enterprise to use its python libraries in our notebooks.Earlier we had issues with downloading our repository using the repos feature in Databricks platform since only notebooks can b...
- 1068 Views
- 1 replies
- 1 kudos
Latest Reply
To fix the issue, we need to pass the token in the header itself git clone https://<token>:x-oauth-basic@github.com/owner/repo.gitExample:%sh
git clone https://<token>@github.com/darshanbargal4747/databricks.git
- 677 Views
- 1 replies
- 0 kudos
while connecting from notebook, it returns the error unable to resolve name.
- 677 Views
- 1 replies
- 0 kudos
Latest Reply
Since we are unable to resolve hostname, it point towards the DNS issue. We can use custom dns using init script and add in the cluster:%scala
dbutils.fs.put("/databricks/<directory>/dns-masq.sh";,"""
#!/bin/bash
#####################################...
- 505 Views
- 0 replies
- 0 kudos
How can I verify if auto-optimize is activated from Delta history for the two scenarios below? Will the DESC history show the details in both the cases? 1). Auto-optimize set on the table properties2). Auto-optimize enabled in spark sessionP.S. - I'm...
- 505 Views
- 0 replies
- 0 kudos
- 856 Views
- 1 replies
- 0 kudos
we have automated out deployment with python API's however we have been caught in a situation which we cannot yet solve.We are looking to collect a token during the first deployment within the environment. currently our API requires a token.Is there...
- 856 Views
- 1 replies
- 0 kudos
Latest Reply
We can use below API to create a token and use the username and passwordcurl -X POST -u "admin_email":"xxxx" https://host/api/2.0/token/create -d'
{
"lifetime_seconds": 100,
"comment": "this is an example token"
}'
- 5910 Views
- 1 replies
- 1 kudos
Also curious if you can export a notebook created in Databricks as a Jupyter notebook
- 5910 Views
- 1 replies
- 1 kudos
Latest Reply
Yes, the .ipynb format is a supported file type which can be imported to a Databricks workspace. Note that some special configurations may need to be adjusted to work in the Databricks environment. Additional accepted file formats which can be import...
- 1050 Views
- 1 replies
- 0 kudos
Where do I start when starting performance tuning of my queries? Are there particular things I should be looking out for?
- 1050 Views
- 1 replies
- 0 kudos
Latest Reply
Few things on top of my mind.1) Check Spark UI and check which stage is taking more time.2) Check for data skewing3) Data skew can severely downgrade performance of queries, Spark SQL accepts skew hints in queries, also make sure to use proper join h...
- 437 Views
- 1 replies
- 0 kudos
Wondering if I can make any kind of custom visuals or are the ones that come built in the only options?
- 437 Views
- 1 replies
- 0 kudos
Latest Reply
At this time the only available visuals are the ones that are included in the Databricks SQL environment. There is no way to import or create custom visuals.
- 1512 Views
- 1 replies
- 0 kudos
Does the query have to be re-run from the start, or can it continue? Trying to evaluate what risk there is by using spot instances for production jobs
- 1512 Views
- 1 replies
- 0 kudos
Latest Reply
If a spot instance is reclaimed in the middle of a job, then spark will treat it as a lost worker. The spark engine will automatically retry the tasks from the lost worker on other available workers. So the query does not have to start over if indivi...
- 443 Views
- 0 replies
- 0 kudos
I have created Delta tables in my Databricks workspace and would like to access them using Microstrategy. Is this possible?
- 443 Views
- 0 replies
- 0 kudos
