- 1624 Views
- 1 replies
- 0 kudos
Error ingesting files with databricks jobs
The source path that i want to ingest files with is:"gs://bucket-name/folder1/folder2/*/*.json"I have a file in this path that ends with ".json.gz" and the databricks job ingests this file even though it doesn't suppose to.How can i fix it?Thanks.
- 1624 Views
- 1 replies
- 0 kudos
- 0 kudos
Thanks Kaniz
- 0 kudos
- 9632 Views
- 0 replies
- 0 kudos
Deleted the s3 bucket assocated with metastore
I deleted the aws s3 bucket for the databricks metastore by mistake.How to fix this? can I re-create the s3 bucket? Or can I delete the metastore (I don't have much data in it), and re-generate one? Thank you!
- 9632 Views
- 0 replies
- 0 kudos
- 14154 Views
- 2 replies
- 1 kudos
Resolved! Not able to access Account console page
Hello All,I'm not be able to access the account console page. I'm a portal admin, my workspace is premium, and yet the Databricks portal stays in a loop, always returning to the Workspaces Overview page and not going to the accounts console page so t...
- 14154 Views
- 2 replies
- 1 kudos
- 1 kudos
Guys, the problem is that I was not signed in as a Global Administrator in Azure AD. After that, I reset all the Browser settings and I was able to do that. Here's the tip.
- 1 kudos
- 1988 Views
- 1 replies
- 1 kudos
Performance issue while calling Sagemaker Endpoint in pyspark udf
Hi,I have pyspark dataframe which calls pyspark udf which in turn calls sagemaker endpoint. But when dataframe has more rows, endpoint start failing. Also it takes longer to process.Please suggest how to call sagemaker endpoint from pyspark.Regards,S...
- 1988 Views
- 1 replies
- 1 kudos
- 2030 Views
- 0 replies
- 0 kudos
org.apache.spark.SparkException - FileReadException
Sometimes getting this kind of error "org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 12224.0 failed 4 times, most recent failure: Lost task 1.5 in stage 12224.0 (TID ) (12.xxx.x.xxx executor 1): com.datab...
- 2030 Views
- 0 replies
- 0 kudos
- 3568 Views
- 2 replies
- 0 kudos
how to edit or delete the post in this community
how to edit or delete the post in this community
- 3568 Views
- 2 replies
- 0 kudos
- 3942 Views
- 3 replies
- 1 kudos
Add Oracle Jar to Databricks cluster policy
I created a policy for users to use when they create their own Job clusters. When I'm editing the policy, I don't have the UI options for adding library (I can only see Definitions and Permissions tabs). I need to add via JSON the option to allows th...
- 3942 Views
- 3 replies
- 1 kudos
- 1 kudos
@adrianhernandez are you admin to workspace, if not you might be missing permissions, if you have policies enabled, admin can allow you.https://docs.databricks.com/en/administration-guide/clusters/policies.html#librariesif your workspace is Unity cat...
- 1 kudos
- 1998 Views
- 2 replies
- 1 kudos
Web terminal and clusters
Hi, I have come across this piece of documentation:Databricks does not support running Spark jobs from the web terminal. In addition, Databricks web terminal is not available in the following cluster types:Job clustersClusters launched with the DISAB...
- 1998 Views
- 2 replies
- 1 kudos
- 3411 Views
- 0 replies
- 0 kudos
Creating java UDF for Spark SQL
Hello, I have created a sample java UDF which masks few characters of a string. However I facing couple of issues when uploading and using it.First I could only import it, which for now is OK. But when do the following,create function udf_mask as 'ba...
- 3411 Views
- 0 replies
- 0 kudos
- 2666 Views
- 0 replies
- 2 kudos
dbutils.fs.ls MAX_LIST_SIZE_EXCEEDED
Hi!I'm experiencing different behaviours between two DBX Workspaces when trying to list file contents from an abfss: location.In workspace A running len(dbutils.fs.ls('abfss://~~@~~~~.dfs.core.windows.net/~~/')) results in "Out[1]: 1551", while runni...
- 2666 Views
- 0 replies
- 2 kudos
- 22511 Views
- 3 replies
- 4 kudos
Training @ Data & AI World Tour 2023
Join your peers at the Data + AI World Tour 2023! Explore the latest advancements, hear real-world case studies and discover best practices that deliver data and AI transformation. From the Databricks Lakehouse Platform to open source technologies in...
- 22511 Views
- 3 replies
- 4 kudos
- 4 kudos
Introducing Mini Flush: Your Ticket to Ultimate Casino Thrills!Are you ready to embark on an electrifying journey into the world of online gambling? If so, look no further than Vijaybet Online Casino! Our state-of-the-art platform is your gateway to ...
- 4 kudos
- 2493 Views
- 1 replies
- 1 kudos
Resolved! Problem creating external delta table on non-AWS s3 bucket
I am testing Databricks with non-AWS S3 object storage. I can access the non-AWS S3 bucket by setting these parameters:sc._jsc.hadoopConfiguration().set("fs.s3a.access.key", "XXXXXXXXXXXXXXXXXXXX")sc._jsc.hadoopConfiguration().set("fs.s3a.secret.key...
- 2493 Views
- 1 replies
- 1 kudos
- 1 kudos
Found the solution to disable it. Can close this question.
- 1 kudos
- 4931 Views
- 3 replies
- 1 kudos
getArgument works fine in interactive cluster 10.4 LTS, raises error in interactive cluster 10.4 LTS
Hello,I am trying to use the getArgument() function in a spark.sql query. It works fine if I run the notebook via an interactive cluster, but gives an error when executed via a job run in an instance Pool.query:OPTIMIZE <table>where date = replace(re...
- 4931 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Retired_mod,Would you be able to respond to my last comment? I couldn't manage to get it working yet.Thank you in advance.
- 1 kudos
- 28594 Views
- 0 replies
- 3 kudos
Schema owned by Service Principal shows error in PBI
Background info:1. We have unity catalog enabled. 2. All of our jobs are run by Service Principal that has all necessary access it needs.Issue:One of the jobs checks existing schemas against the ones it is supposed to create in that given run and if ...
- 28594 Views
- 0 replies
- 3 kudos
- 3190 Views
- 1 replies
- 0 kudos
AWS Databricks VS AWS EMR
HiWhich services should I use for data lake implementation?any cost comparison between Databricks and aws emr.which one is best to choose
- 3190 Views
- 1 replies
- 0 kudos
- 0 kudos
@AH that depends on use case, if your implementation involves Data Lake, ML, Data engineering tasks better to go with databricks as it has got good UI and there good governance using unity catalog for your data lake and you have good consumer tool su...
- 0 kudos
