unable to create delta tables in aws glue catalogThe project requires that we integrate with the AWS Glue catalog.We would like to be able to create tables in delta format in the glue catalog.To test this functionality. We did the followingCreated th...
When I attempt to save my username and token for Github I receive a “Failed to Save. Try again.” message. I’ve used Azure DevOps with another DB workspace and never had an issue saving my PAT. I’ve tried using both my GitHub username and email wi...
Quick update that I’ve now attempted to save my PAT for Github using two different computers and browser types (Safari and Chrome) and all have given the same “Failed to save. Please try again” message. Thankfully I can still clone from public repo...
I am trying to write data from databricks to an S3 bucket but when I submit the code, it runs and runs and does not make any progress. I am not getting any errors and the logs don't seem to recognize I've submitted anything. The cluster also looks un...
The Problem:I've observed erratic behavior when I add a comment containing a trailing escape character (\) to a CREATE TABLE statement.For example, this query returns data (though it shouldn't):CREATE TABLE example_table SELECT 1 -- This comment has ...
@Graham Carman we're tracking this as a defect / issue on our side. For now, please don't include the escape character in comments.
What is the best way to parallelize fbprophet?
I am using managed databricks on gcp. I have 11TB of data with 5B rows. Data from source is not partitioned. I'm having trouble loading the data into dataframe and do further data processing. I have tried couple of executors configuration , none of t...
I am trying to upload blob storage on databricks sql warehouse. I followed this document https://docs.databricks.com/data/data-sources/azure/azure-storage.html. but this doesn't seem to be working. Query executed fine but created schema was empty. An...
@Athar Abbas the simplest thing would be to create a SAS token to the ADLS Gen 2 container and then use the COPY INTO command with the AZURE_SAS_TOKEN credential: https://docs.microsoft.com/en-us/azure/databricks/data/data-sources/azure/adls-gen2/az...
We design and manufacture intelligent products and smart solutions that make your life easier, safer and more comfortable. Via the registered 2USB brand we produce innovative easy to install USB stopcontact & powerful USB charging solutions. We are a...
I'm using Databricks' support for GitHub repos. When I switch from one branch to another while a notebook is open, it messes up my notebook. Specifically, every notebook cell appears twice after switching branches.
How to fix below SSL error while setting up databricks Cli or installing library in clusterLibrary installation attempted on the driver node of cluster *** and failed.Please refer to the following error message to fix the library or contact Databric...
I was getting an SSL when trying to set up secrets using the Databricks CLI. To fix the CLI SSL error I went to %USERPROFILE%\.databrickscfg (~/.databrickscfg on Unix, Linux, or macOS) from the file explorer on my local machine and added the insecure...
Hi,As for now we already know that our application will be running 24/7 streaming constantly incoming data. The stream pipeline is very basic, however as for now it's enough to run this pipeline 1x per day (to save the costs of constantly running clu...
Use .trigger(once=True) or .trigger(availableNow=True) option which will pick only the new files https://docs.databricks.com/structured-streaming/triggers.html#configuring-incremental-batch-processing
Is there a way to create Databricks unmanaged table that's actually Snowflake table, not some S3 or DBFS location?From documentation is rather vague is this possible: "You can create an unmanaged table with your data in data sources such as Cassandra...
Hey @Stefan Stojanovic Hope everything is going great!Does @Hubert Dudek's response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly? Else please let us know if you need mo...
We are converting Pyspark dataframe to Tensorflow using PetaStorm and have encountered a “data adapter” error. What do you recommend for diagnosing and fixing this error?https://docs.microsoft.com/en-us/azure/databricks/applications/machine-learning/...
Hey @Nathan Law Thank you so much for getting back to us. We will await your response.We really appreciate your time.
I have a large delta table partitioned by an identifier column that I now have discovered has blank spaces in some of the identifiers, e.g. one partition can be defined by "Identifier=first identifier". Most partitions does not have these blank space...
FYI similar issue with partitions with "%" in the identifier. Used the filter clause of the REORG to exclude partitions with " " or "%" to be able to move forward with my work but will continue looking for a solution. I've never seen any pointers not...
I wrote the following code:data = spark.sql (" SELECT A_adjClose, AA_adjClose, AAL_adjClose, AAP_adjClose, AAPL_adjClose FROM deltabase.a_30min_delta, deltabase.aa_30min_delta, deltabase.aal_30min_delta, deltabase.aap_30min_delta ,deltabase.aapl_30m...
I just discovered a solution.Today, I opened Azure Databricks. When I imported python libraries. Databricks told me that toPandas() was deprecated and it suggested me to use toPandas.The following solution works: Use toPandas instead of toPandas() da...
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 566 | |
| 349 | |
| 287 |
