- 24005 Views
- 12 replies
- 2 kudos
Resolved! Can't login to databricks community edition
I enter my valid login and password here https://community.cloud.databricks.com/login.html but it says "Invalid email address or password"
- 24005 Views
- 12 replies
- 2 kudos
- 2 kudos
data bricks cluster is terminating each 15 sec in community edition. Could you please help me on this issue.
- 2 kudos
- 5052 Views
- 7 replies
- 6 kudos
Resolved! Some DLTpipelines suddely seem to take different runtime 16.1 instead of 15.4 since last night (CET)
Hello, Suddenly since last night on some of our DLT pipelines we're getting failures saying that our hive_metastore control table cannot be found. All of our DLT's are set up the same (serverless), and one Shared Compute on runtime version 15.4. For ...
- 5052 Views
- 7 replies
- 6 kudos
- 6 kudos
@voo-rodrigo Hello, thanks for updating the progress on your end! I've tested as well and confirmed that the DLT can read the hive_metastore via Serverless again.
- 6 kudos
- 8080 Views
- 6 replies
- 3 kudos
Resolved! Community Edition is not allocating Cluster
I've been trying to use the Community edition for the past 3 days without success.I go to run a Notebook and it begins to allocated the Cluster, but it it never finishes. Sometimes it times out after 15 minutes.Waiting for cluster to start: Finding i...
- 8080 Views
- 6 replies
- 3 kudos
- 3 kudos
I am running into the same issue since today. It worked fine till yesterday.
- 3 kudos
- 3539 Views
- 3 replies
- 0 kudos
How to access system.access.table_lineage
I try to make a select from system.access.table_lineage but i dont have to see the tablewhat permission to i have
- 3539 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @ZacayDaushin,To query the table in system catalog, you need to have SELECT permission on top of the table to query and see the results.Best Regards,Nivethan V
- 0 kudos
- 1998 Views
- 1 replies
- 1 kudos
Resolved! tbl name as paramater marker
I am getting an error here, when I do this//this works fine declare sqlStr = 'select col1 from catalog.schema.tbl LIMIT (?)'; declare arg1 = 500; EXECUTE IMMEDIATE sqlStr USING arg1; //this does not declare sqlStr = 'select col1 from (?) LIMIT (?)';...
- 1998 Views
- 1 replies
- 1 kudos
- 1 kudos
@smpa01 In SQL EXECUTE IMMEDIATE, you can only parameterize values, not identifiers like table names, column names, or database names.That is, placeholders (?) can only replace constant values, not object names (tables, schemas, columns, etc.).SELECT...
- 1 kudos
- 2055 Views
- 4 replies
- 0 kudos
Structured Streaming writeStream - Query is no longer active causes task to fail
Hi, I execute readStream/writeStream in workflow task. Write stream uses .trigger(availableNow=True) option. After writeStream I'm waiting query to finish with query.awaitTermination(). However from time to time, pipeline ends with "Query <id> is no ...
- 2055 Views
- 4 replies
- 0 kudos
- 0 kudos
@Alberto_Umana this bug was apparently fixed a few months ago, but we're still facing the same issue on our end.
- 0 kudos
- 1978 Views
- 1 replies
- 1 kudos
Resolved! Several unavoidable for loops are slowing this PySpark code. Is it possible to improve it?
Hi. I have a PySpark notebook that takes 25 minutes to run as opposed to one minute in on-prem Linux + Pandas. How can I speed it up?It's not a volume issue. The input is around 30k rows. Output is the same because there's no filtering or aggregation...
- 1978 Views
- 1 replies
- 1 kudos
- 1 kudos
@397973 Spark is optimized for 100s of GB or millions of rows, NOT small in-memory lookups with heavy control flow (unless engineered carefully).That's why Pandas is much faster for your specific case now.Pre-load and Broadcast All MappingsInstead of...
- 1 kudos
- 2076 Views
- 1 replies
- 0 kudos
SocketTimeoutException when creating execution context in Databricks Community Edition
Hello,I’m experiencing an issue in Databricks Community Edition.When I try to run a notebook, I get this error:"Exception when creating execution context: java.net.SocketTimeoutException: connect Timeout"What I have tried:- Restarting the cluster- Ch...
- 2076 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @Lo! There is a similar thread where another user encountered the same issue and shared a solution that worked for them. I suggest reviewing that thread to see if the solution is helpful in your case as well.
- 0 kudos
- 2400 Views
- 1 replies
- 1 kudos
Issue reading Vertica table into Databricks - Numeric value out of range
I am trying to read a Vertica table into a Spark DataFrame using JDBC in Databricks.Here is my sample code:hostname = ""username = ""password = ""database_port = ""database_name = ""qry_col_level = f"""SELECT * FROM analytics_DS.ansh_units_cum_dash""...
- 2400 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @vidya_kothavale, based on my research and understanding, Databricks and Spark's JDBC connectors currently don’t offer an automatic way to truncate or round high precision decimal values when loading data. To handle this, you would need to either:...
- 1 kudos
- 3128 Views
- 1 replies
- 0 kudos
DEV and PROD
"SELECT * FROM' data call on my table in PROD is giving all the rows of data (historical data), but a call on my table in DEV is giving me just one row of data (current one row of historical data). what could be the problem??
- 3128 Views
- 1 replies
- 0 kudos
- 0 kudos
Please don't cross post. Thanks, Louis.
- 0 kudos
- 3332 Views
- 6 replies
- 1 kudos
Resolved! GET /api/2.2/jobs/list Ordering
Hi there!I am calling the job list API (via the Python SDK):GET /api/2.2/jobs/listdocs.databricks.com/api/workspace/jobs/listDoes anyone know what ordering is applied / calculated for the list of jobs? Is it consistent or random?Is it by creation tim...
- 3332 Views
- 6 replies
- 1 kudos
- 3089 Views
- 7 replies
- 0 kudos
Google Pub Sub and Delta live table
I am using delta live table and pub sub to ingest message from 30 different topics in parallel. I noticed that initialization time can be very long around 15 minutes. Does someone knows how to reduced initialization time in dlt ? Thanks You
- 3089 Views
- 7 replies
- 0 kudos
- 0 kudos
Classic clusters can take up to seven minutes to be acquired, configured, and deployed, with most of this time spent waiting for the cloud service to allocate virtual machines. In contrast, serverless clusters typically start in under eight seconds. ...
- 0 kudos
- 1997 Views
- 3 replies
- 2 kudos
Resolved! How can we get AutoLoader to detect a file footer?
We are dealing with CSVs that have footers in them. When we have an empty file, the presence of this footer seems to impair the schema inferencing of AutoLoader, because of the footer.I know where is a header = true parameter, but I don't see a foote...
- 1997 Views
- 3 replies
- 2 kudos
- 2 kudos
To be clear, when you say footer are you referring to the last row of the tuple? e.g. Header = row 1, Footer = row_last.
- 2 kudos
- 1551 Views
- 2 replies
- 1 kudos
Resolved! Can we implement Unity Catalog table lifecycle?
I want to delete tables that haven't been selected or otherwise accessed for several months.I can see the Delta table history, but I can only catch the DDL or update/insert/delete and can't catch "select".I realized that the Unity Catalog insight, ht...
- 1551 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Renu_ ,I appreciate for your clear response. I now have a better understanding and will work with our admin team to develop a strategy.Thank you.
- 1 kudos
- 3078 Views
- 2 replies
- 0 kudos
Resolved! Concurency behavior with merge operations
Hi community,I have this case right now in project where i have to develop a solution that will prevent duplicate data from being ingested twice to delta lake. Some of our data suppliers at a rare occurence are sending us the same dataset in two diff...
- 3078 Views
- 2 replies
- 0 kudos
- 0 kudos
Your idea of using a log table to track processed ingestions and leveraging a MERGE operation in your pipeline is a sound approach for preventing duplicate data ingestion into Delta Lake. Delta Lake's ACID transactions and its support for concurrency...
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 582 | |
| 349 | |
| 287 |
